Method of utilizing defocus in virtual reality and augmented reality

ABSTRACT

The current invention relates to the methods to achieve virtual reality with making virtual objects not under focus of a viewer in virtual scenes appear defocused, and making objects under focus of said viewer appearing focused and clear to said viewer. The current invention also relates to the methods to achieve augmented reality with making virtual objects in virtual scenes associated with real objects not under focus of a viewer in real scenes appear defocused to said viewer in said virtual scene, and making virtual objects in virtual scenes associated with real objects being under focus of a viewer in real scenes appear focused to said viewer in said virtual scene.

BACKGROUND

The current invention generally relates to three-dimensional visualperception technology and more particularly to a system and method forrealizing real-time re-focusable stereo vision.

Stereo vision, or stereoscopic vision, better known as 3D vision,realizes three-dimensional visual perception of an object by recordingthe images of the same object from at least two different viewingangles, and then displaying the two different images separately to eachof the two eyes of a viewer. The viewer perception from the separatelyshown images of the same object to different eyes with different viewingangles is a three-dimensional object existing in the viewer's viewingspace.

For motion picture utilizing stereo vision, i.e. 3D movies, imagerecording by the recording devices generally has only a single focusdepth. The objects not being focused upon by the recording devices stayde-focused in the recorded images and are perceived as blurred objectsto the viewer during stereoscopic playback of the 3D movies. In priorart practices of 3D recording and viewing, a viewer is not given theability to re-focus on the defocused and blurred objects as one can doin reality.

For a 3D viewing experience of the viewer to better simulate a real-lifethree-dimensional visualization of objects within the viewing space ofthe viewer, it is desirable for a viewer to be able to focus on theobjects of interest and be able to re-focus on new objects within thesame viewing space, following viewer's own re-focusing intention, forexample by viewer's eye-lens change, eyeball position change orbrain-wave pattern change that naturally happen during a human visionre-focus event without viewer's active effort to change the focus depthof the shown images. Thus a reality viewing experience can be achieved.The ability of being able to focus on objects of interest by viewer'sintention, without active effort from viewer, during stereo vision,gives unprecedented advantage in its closest-to-reality viewingexperience. This ability will promote stereo vision's application inareas where varying focus depth vision provides best life-like visualcomprehension of an object of interest.

As shown in FIG. 1, vision of a human eye 11 is achieved by three keyoptical components that determine the imaging of surrounding objectsthat the eye can see: the Lens (eye-lens) 1, the Retina 2, and the Iris3. The lens 1 is the component that functions the same as the opticallenses used in cameras. Light reflected or emitted from an outsideobject passes through the pupil 9 and the lens 1. An optical image ofthe object is projected on the retina 2 with the light from the objectbeing re-focused by the lens 1. The lens 1 is controlled by the CiliaryMuscle 4 and Ligament 5 which can compress or stretch the lens 1 shape,which in turn changes the optical focus depth of the lens 1 and makesobjects at various distances from the viewer producing focused images onthe retina 2, and thus the viewer can see objects far or near clearly.This control of lens focus depth gives a viewer the ability to seeobjects near and far at will. The retina 2 is like a film screen withina camera. When the light from an object passes through the lens land isprojected onto the retina 2 and makes a clear and focused image, thevision cells of the retina 2 sense the color and intensity of theprojected image and send such information to human brain through theoptical nerves 6, and thus human vision is realized. The iris 3 controlsthe total amount of light that can go into the eye by adjusting thepupil 9 size, which helps maintain the right amount of light intensitythat goes into the eye 11 without damaging the retina cells.

FIG. 2 is a schematic diagram illustrating how normal human vision isachieved according to prior art. Same object 29 is projected intodifferent images 25 and 26 in different eyes 21 and 22 of a viewer dueto the angle of viewing is different for the two eyes 21 and 22. Theangle difference as inferred from the two images 25 and 26 of the sameobject 29 in the two eyes 21 and 22 as being perceived by the brain isused to extract the information as to how far the object 29 is from theviewer. When images of the same object 29 are taken at different viewingangles, and then projected separately onto the retina 24 of thedifferent eyes 21 and 22 of a viewer, the viewer can also have a similardistance perception of the object in the viewing space, where the objectis actually not existent. This gives rise to the stereo vision, or 3Dvision, meaning viewing of an object with a distance perception from theviewer.

FIG. 3 is a schematic diagram illustrating stereo-vision being achievedaccording to prior art. The principle function of all currently existingstereo-vision, or 3D vision, is the same, which includes: (1) Projectingtwo different images 391 and 392 of the same object 390 (not shown inFIG. 3) captured at two different angles on the same screen 38; (2)Allowing each eye 21 and 22 to see only one of the two images 391 and392; and (3) The viewer with each eye 21 and 22 seeing a differentprojected image 25 and 26 from images 391 and 392 taken at differentangle of the same object 390 perceives an imaginary object 39 in spacethat is at a distance from viewer different than the screen 38 whereimages 391 and 392 are shown.

When the stereo-vision is applied to a motion picture, a 3D movie isproduced. The methods used to achieve each eye viewing different imagesare accomplished by wearing 3D viewing glasses that can do any of: (1)filter polarized light; (2) filter light of different colors; and (3)have timed shutter being synchronized with the showing of differentviewing angle images on the screen. By showing the images of the sameobject recorded at different angles, arranging the images at differentlocations on the same screen, and using a method to individually showimage recorded at different view angle to different eye, viewerperceived an imaginary object in space at a distance from the viewerdifferent than the screen distance to the viewer.

FIG. 4 is a schematic diagram illustrating the problems of the prior artstereo-vision techniques. A fundamental drawback of all existingstereo-vision techniques and 3D movie techniques in the attempt tosimulate real-life viewing experience is that when the object images arecaptured from two different viewing angles, objects 391 and 392 that arefocused upon will show up as focused objects when projected on screen.Objects 491 and 492 within the same scene but not focused upon duringrecording will stay defocused on the screen 38. Thus, when viewerperceived the 3D image, only the objects 391 and 392 that are focusedupon during image capturing can be viewed clearly, while other objects491 and 492 stay blurred. Viewer only sees a clear imaginary object 39from the images 391 and 392, while object 49 from images 491 and 492 aredefocused. The existing prior art techniques do not allow viewer to viewall objects within same scene clearly and does not have a method tobring objects into focus at viewer's own discretion. Even though otherobjects in the recorded images also show up on the same screen 38, dueto the fact that the focus was only on the object where images 391 and392 are taken from, other objects stay defocused. Thus, viewer'sintention of focusing upon the objects 491 and 492 that are notcurrently in-focus cannot be achieved in conventional prior art stereovision. This limitation makes 3D vision of prior art an obviousdeviation from reality. In comparison, in real life, for objects near orfar, a viewer can freely adjust to their distance with eye-lens focallength change and eyeball pupil position change and achieve clear viewof any object in the viewing angle. Prior art is limited in the abilityto re-produce the life-like stereo-vision viewing experience.

It is desired to have a method and an apparatus that can achievereal-time re-focusable vision based on viewer's own re-focus intentionto simulate more life-like stereo-vision experience without activeviewer participation or intervention.

SUMMARY OF THE INVENTION

This invention proposes a novel method to realize the real-timere-focusable stereo vision with utilizing: (1) producing stereoscopicimages at multiple focus depth; (2) active sensing the intention ofvision re-focus from the viewer; and (3) retrieving and displaying ofthe images with the focus depth according to sensed viewer-desired focusdepth in real time to produce stereo vision to the viewer, whichreflects viewer's intended focus depth with objects of interest beingin-focus in viewer's vision.

This method provides the viewer the ability to view the objects ofinterest in focus at will, while not requiring the viewer's effort toactively participate to achieve such re-focus task.

This invention helps achieve 3D vision that most closely simulatesreal-life viewing experience, and can give the impression of viewing areal-life scene where viewer can focus on objects of interest with pureintention.

Viewer intention of re-focused is obtained by sensing and calculatingthe natural changes of the eye-lens shape and/or curvature, eye ballpupil position, or by brain-wave pattern.

Although this invention is intended to achieve re-focusable stereovision, same techniques can also be used to achieve re-focusablenon-stereoscopic flat vision without limitation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustrating human eye's

FIG. 2 is a schematic illustrating human vision being formed accordingto prior art.

FIG. 3 is a schematic illustrating stereo-vision being formed accordingto prior art.

FIG. 4 is a schematic illustrating the limitation of the prior artstereo-vision techniques.

FIG. 5 is a schematic illustrating the first implementation for the stepof recording of same scene simultaneously into multiple images with eachimage recorded with a different focus depth into the scene according tothe embodiments of the current invention.

FIG. 6 is a schematic illustrating the second implementation for thestep of recording of same scene simultaneously into multiple images witheach image recorded with a different focus depth into the sceneaccording to the embodiments of the current invention.

FIG. 7 is a schematic illustrating the third implementation for the stepof recording of same scene simultaneously into multiple images with eachimage recorded with a different focus depth into the scene according tothe embodiments of the current invention.

FIG. 8 is a schematic illustrating the fourth implementation for thestep of recording of same scene simultaneously into multiple images witheach image recorded with a different focus depth into the sceneaccording to the embodiments of the current invention.

FIG. 9 is a schematic illustrating the first implementation for the stepof sensing the re-focus intention of viewer according to the embodimentsof the current invention.

FIG. 10 is a schematic illustrating the second implementation for thestep of sensing the re-focus intention of viewer according to theembodiments of the current invention.

FIG. 11A is a schematic illustrating the third implementation for thestep of sensing the re-focus intention of viewer according to theembodiments of the current invention.

FIG. 11B is a schematic illustrating the fourth implementation for thestep of sensing the re-focus intention of viewer according to theembodiments of the current invention.

FIG. 12A is a schematic illustrating the fifth implementation for thestep of sensing the re-focus intention of viewer according to theembodiments of the current invention.

FIG. 12B is a schematic illustrating the scanning procedure of theprobing light 1241 by oscillating the optical emitter 124 of FIG. 12A.

FIG. 12C is a schematic illustrating the sixth implementation for thestep of sensing the re-focus intention of viewer according to theembodiments of the current invention.

FIG. 12D is a schematic illustrating the scanning procedure of theprobing light 1241 by oscillating the scanning mirror or prism 1242 ofFIG. 12C.

FIG. 12E is a schematic illustrating one example of optical signalssensed by the optical detector 125 during scanning of the probing light1241 in FIG. 12B and FIG. 12D.

FIG. 12F is a schematic illustrating the pupil positions of the eyes ofthe viewer when viewer is focusing on a far point.

FIG. 12G is a schematic illustrating the pupil positions of the eyes ofthe viewer when viewer is focusing on a near point.

FIG. 12H is a schematic illustrating an example of utilizing FIG. 12Aimplementation to sense a viewer's re-focus intention.

FIG. 13A is a schematic illustrating the seventh implementation for thestep of sensing the re-focus intention of viewer according to theembodiments with using a contact-lens type of see-through substratebeing used for the same purpose of the see-through substrates in theimplementations of FIG. 9, FIG. 10, FIG. 11A, and FIG. 11B.

FIG. 13B is a schematic illustrating a see-through substrate withembedded circuitry, optical emitter and optical detector for detectionof change of focus depth of the eye.

FIG. 13C is a schematic illustrating a contact-lens type of see-throughsubstrate with embedded circuitry, optical emitter and optical detector,working together with a fixed frame in proximity to the eye, fordetection of change of focus depth of the eye.

FIG. 14 is a schematic illustrating the eighth implementation for thestep of sensing the re-focus intention of viewer according to theembodiments of the current invention.

FIG. 15 is a schematic illustrating the ninth implementation for thestep of sensing the re-focus intention of viewer according to theembodiments of the current invention.

FIG. 16 is a schematic illustrating the second option for the step ofdisplaying the retrieved image according to the embodiments of thecurrent invention.

FIG. 17 is a schematic illustrating the third option for the step ofdisplaying the retrieved image according to the embodiments of thecurrent invention.

FIG. 18 is a schematic flow diagram illustrating the first embodimentwherein eye-lens and eye-ball sensing of eye-information are used.

FIG. 19 is a schematic flow diagram illustrating the first embodimentwherein brain-wave pattern sensing of re-focus intention are used.

FIG. 20 is a schematic flow diagram illustrating the second embodimentwherein eye-lens and eye-ball sensing of eye-information are used.

FIG. 21 is a schematic flow diagram illustrating the second embodimentwherein brain-wave pattern sensing of re-focus intention are used.

FIG. 22 is a schematic flow diagram illustrating the third embodimentwherein eye-lens and eye-ball sensing of eye-information are used.

FIG. 23 is a schematic flow diagram illustrating the third embodimentwherein brain-wave pattern sensing of re-focus intention are used.

FIG. 24 is a schematic flow diagram illustrating the fourth embodimentwherein eye-lens and eye-ball sensing of eye-information are used.

FIG. 25 is a schematic flow diagram illustrating the fourth embodimentwherein brain-wave pattern sensing of re-focus intention are used.

FIG. 26 is a schematic of a flow diagram illustrating a feed-back loopduring re-focus.

FIG. 27 is a schematic illustrating application of invention in pictureson display screen.

FIG. 28 is a schematic illustrating application of invention in picturesby image projector.

FIG. 29 is a schematic illustrating application of invention forenhanced vision.

FIG. 30 is a schematic illustrating application of invention forartificial reality.

FIG. 31A and FIG. 31B are schematics illustrating the application of theinvention for augmented reality with artificial object augmentinginteraction with real objects.

FIG. 32 is a schematic illustrating the application of the invention foraugmented reality with artificial object augmenting real objects.

FIG. 33 is a schematic illustrating the application of the invention foraugmented reality with artificial object augmenting viewer's interactionwith real objects.

FIG. 34 is a schematic illustrating a MEMS-based micro-mirror array usedfor direct projection of image on the retina of viewer's eye.

FIG. 35 is a schematic illustrating the MEMS-based micro-mirror array ofFIG. 34 being implemented with input from viewer's eye information toaccommodate the viewer's eye-lens change and project image in focus onretina at varying eye-lens focus depth, with reflected light convergingtowards an area inside the eye-lens.

FIG. 36A is a schematic illustrating the MEMS-based micro-mirror arraybeing implemented with reflected light converging towards an area behindthe eye-lens before entering the pupil of the eye.

FIG. 36B is a schematic illustrating the MEMS-based micro-mirror arraybeing implemented with reflected light converging towards an area infront of the eye-lens before entering the pupil of the eye.

FIG. 37A is a schematic illustrating a substrate with embedded array oflight emitting devices for the purpose of projecting image on the retinaof the eye.

FIG. 37B is a schematic illustrating the substrate of FIG. 37A withembedded array of light emitting devices having a first type of lightbeam shaping components.

FIG. 37C is a schematic illustrating the substrate of FIG. 37A withembedded array of light emitting devices having a second type of lightbeam shaping components.

FIG. 38 is a schematic illustrating a substrate with embedded array oflight passage components used to project image on retina with incominglight shining towards the eye.

FIG. 39A is a schematic illustrating a substrate with embedded array oflight passage components for the purpose of projecting image on theretina of the eye.

FIG. 39B is a schematic illustrating the substrate of FIG. 39A withembedded array of light passage components having a first type of lightbeam shaping components.

FIG. 39C is a schematic illustrating the substrate of FIG. 39A withembedded array of light passage components having a first type of lightbeam shaping components.

FIG. 40A is a schematic illustrating how a flexible substrate withembedded array of light emitting devices is used to project image on theretina at a first focus depth of the eye-lens.

FIG. 40B is a schematic illustrating the flexible substrate of FIG. 40Awith embedded array of light emitting devices is used to project imageon the retina at a second focus depth of the eye-lens with change ofcurvature and position of the flexible substrate.

FIG. 41A is a schematic illustrating embodiment how flexible substrateswith embedded array of light emitting devices, or array of light passagecomponents, can be used to project image on the retina of the eye whenthe viewer is focusing on various objects at a first distance and afirst viewing angle relative to the viewer.

FIG. 41B is a schematic illustrating how flexible substrates of FIG. 41Awith embedded array of light emitting devices, or array of light passagecomponents, project image on the retina of the eye when the viewer isfocusing on objects at a second distance and a second viewing anglerelative to the viewer, with change of the curvature and position of theflexible substrates.

FIG. 41C is a schematic illustrating how flexible substrates of FIG. 41Awith embedded array of light emitting devices, or array of light passagecomponents, project image on the retina of the eye when the viewer isfocusing on objects at a third distance and a third viewing anglerelative to the viewer, with change of the curvature and position of theflexible substrates.

FIG. 42A is a schematic illustrating light beam angle adjustment with aMEMS device.

FIG. 42B is a schematic illustrating light beam angle adjustment withmagnet and electric coil.

FIG. 42C is a schematic illustrating light beam angle adjustment withcapacitive device.

FIG. 42D is a schematic illustrating light beam angle adjustment with apiezo element.

FIG. 42E is a schematic illustrating light beam angle adjustment with athermally induced shape change element.

For purposes of clarity and brevity, like elements and components maybear the same designations and numbering throughout the Figures, whichare not necessarily drawn to scale.

DETAILED DESCRIPTION OF THE INVENTION

While the current invention may be embodied in many different forms,designs or configurations, for the purpose of promoting an understandingof the principles of the invention, reference will be made to theembodiments illustrated in the drawings and specific language will beused to describe the same. It will nevertheless be understood that nolimitation or restriction of the scope of the invention is therebyintended. Any alterations and further implementations of the principlesof the invention as described herein are contemplated as would normallyoccur to one skilled in the art to which the invention relates.

The first embodiment of the current invention is for static or motionpictures. The method according to the first embodiment includes thesteps of: (Step 101) Recording of the same scene simultaneously intomultiple images with each image recorded with a different focus depthinto the scene on recording media; (Step 102) Active sensing there-focus intention of viewer by monitoring the physiological change ofviewer's eye including eye-lens change without viewer's activeparticipation or physical action and generating such physiologicalchange information; (Step 103) Calculating intended focus depth and/orintended in-focus objects in the scene from the physiological changeinformation from Step 102; (Step 104) Retrieving the images withintended focus depth from the recording media containing recorded imagesfrom Step 101; and (Step 105) Displaying the retrieved image from Step104 to the viewer's eyes.

In Step 102, the said physiological change of viewer's eye can alsoinclude the rotational position of the viewer's eye pupil.

In Step 105, an optical imaging system 18912 with a variable effectivefocus depth can be disposed in the optical path between the image andthe viewer's eye, where the effective focus depth of the system isautomatically adjusted to the viewer's eye-lens focus depth change inreal-time according to the physiological change information from Step102, such that the image of Step 105 shown on the same screen appearsfocused on the retina of the viewer's eye at various viewer's eye-lensfocus depth. Such optical image system can be any of: a single opticallens with mechanical positioning, a series or an array of optical lenseswith mechanical positioning, a variable focus depth optical componentthat is composed of electrically-controlled refractive index material,an optical component whose effective optical path for light passingthrough can be changed by an electrical signal, and an optical componentcomposed of micro-electro-mechanical-system (MEMS) actuated lens, mirroror prism arrays that performs effectively as an optical lens or anoptical concave or convex mirror.

FIG. 5 is a schematic diagram illustrating a first implementation forthe step of recording of same scene simultaneously into multiple imageswith each image recorded with a different focus depth into the sceneaccording to the embodiments of the current invention. In thisimplementation, light splitters 54 and 55 are used to split incominglight 51 from scene into different light paths to realize differentfocus depth image recording on different recording media 572, 582 and592. The light splitters 54 or 55 can be any of: a light splitter, aprism, a lens, or a shutter with mirror on light incoming side andshutter can be timed open and close to pass and reflect light. The lightsplitter 54 is preferably positioned at the phase plane of the objectivelens 52. Both the objective lens 52 and the imaging lens 571, 581 or591, can each be composed of a series of lenses. On different recordingmedia 572, 582 and 592, images 500 of different objects of 50 atdifferent distance from the objective lens 52 from the scene are infocus in different light paths 573, 583 and 593. Distance betweenrecording media and imaging lens can be different in different lightpath 573, 583 and 593. Light splitter 54 and 55 transmission andreflection efficiencies may be different for different light path 573and 583. Light reflector 56 provides total light reflection for the lastlight path 593. Different recording media 572, 582 and 592 may havedifferent sensitivity to light intensity to adjust to the differentincoming light intensity of each light path 573, 583 and 593. Imaginglens 571, 581 and 591 of different light path may have different focusdepth and optical property.

There can be more than two light paths in the system and more than twolight splitters accordingly. The two or more of similarly structuredimaging system can be used to record multiple-focus-depth images of samescene in different viewing angles for stereo-vision purpose.

FIG. 6 is a schematic diagram illustrating the second implementation forthe step of recording of same scene simultaneously into multiple imageswith each image recorded with a different focus depth into the scene. Inthis implementation, phase diverter 64 is used to divert incoming light61 from scene into different light paths at different directions torealize different focus depth image recording on different recordingmedia 672, 682 and 692. The phase diverter 64 can be any of: a lensarray, a mirror array, a phase plate, or a phase plate array, which canbe mechanically or electrically actuated. The phase diverter 64 ispositioned at the phase plane (focus plane) of the objective lens 62.Both the objective lens and the imaging lens 671, 681 and 691 can eachbe composed of a series of lens. On different recording media 672, 682and 692, images 600 of different objects 60 at different distance fromthe objective lens 62 from the scene are in focus in different lightpaths 673, 683 and 693. Different recording media 672, 682 and 692 mayhave different sensitivity to light intensity to adjust to the differentincoming light intensity of each light path 673, 683 and 693. Distancebetween recording media 672, 682 or 692, and imaging lens 671, 681 or691 can be different in different light path. Imaging lens 671, 681 and691 may have different focus depth and optical property.

There can be more than three light paths in the system and more thanthree phase diverting paths accordingly. Two or more of similarlystructured imaging system can be used to record multiple-focus-depthimages of same scene in different viewing angles for stereo-vision.

With phase diverter 64 being a two-dimensional lens matrix at the phaseplane of the objective lens 62, same scene may be recorded at differentviewing angles simultaneously by a single system for stereo-visionpurpose, i.e. multiple focus depth and multiple viewing angles recordingcan be achieved at same time.

FIG. 7 is a schematic diagram illustrating the third implementation forthe step of recording of same scene simultaneously into multiple imageswith each image recorded with a different focus depth into the scene. Inthis implementation, shuttered recording media 75 is used to recordimage of scene 700 with different distance objects in-focus on differentrecording media. Single light path with multiple recording media 75arranged in an array along the light path. Both the objective lens 72and the imaging lens 74 can each be composed of a series of lens.Recording media 75 are shuttered to open or close to allow the light 71to go through or to record the image. At any instant time only one mediais recording the image 700. When a recording media is receiving theincoming light and recording the image 700, all media behind will notrecord image and all media in front of the recording media will beshuttered open. After a media finishes recording, it can be shutteredopen or a media in the front is shuttered close to allow another mediato record image 700. On different recording media, images 700 fromdifferent objects 70 at different distance from the objective lens fromthe scene are in focus. Different recording media may have differentsensitivity to light intensity to adjust to the different incoming lightintensity.

Single light path of FIG. 7 can also be realized by having a singlerecording media 75 that moves from a position close to the imaging lens74 and away to a position farther away from imaging lens 74, or moves inthe reversed direction. During the moving process of the media 75, theimages 700 of the objects 70 are captured by the recording media 75 atdifferent distance from the imaging lens. To avoid overlapping therecorded images, image recorded by media 75 at different distance fromlens 74 is constantly removed and stored and media 75 is refreshed.Alternatively, when one image of 700 is recorded by 75, before new imageis recorded, the recorded image is transformed into data stream andstored in digital format in a separate data storage device.

Two or more of similarly structured imaging system can be used to recordmultiple-focus-depth images of same scene in different viewing anglesfor stereo-vision purpose.

FIG. 8 is a schematic diagram illustrating the fourth implementation forthe step of recording of same scene simultaneously into multiple imageswith each image recorded with a different focus depth into the scene. Inthis implementation, a light field recording device 83 is used to recordimage of scene with various focus depth and having different distanceobjects 80 in-focus. Multiple light field recording device can be usedto record image of the scene at various viewing angles. Same light fieldrecording device may be able to record same scene at different viewingangles, achieving multiple focus depth and stereoscopic recording atsame time.

FIG. 9 is a schematic diagram illustrating the first implementation forthe step of sensing the re-focus intention of viewer according to theembodiments of the current invention. This implementation is forretrieving the viewer's eye-information by retina-reflected probinglight. See-through substrate 93 can be any type of substrate that allowsvisible light to pass through.

Please note that in the specifications of various embodiments of thecurrent invention, the word “glass” is sometimes used as one type of, oran implementation of, or interchangeably as, a “see-through substrate”.See-through substrate 93 provides a supporting frame for the transmitter95 and the detector 94 and allow viewer's eye 90 to see through. Lightfrom images that are displayed to the viewer can pass throughsee-through substrate 93 and forms optical projection on the retina 92.See-through substrate 93 can serve as part of the stereo vision systemthat helps images taken from same scene at different viewing anglesbeing shown to each eye 90 of the viewer separately, so that viewer hasa stereo vision impression. The detector 94 and transmitter 95 do notaffect viewer's ability to see the images displayed to the viewer.

Transmitter 95 produces light beam or light pattern that is projectedinto the viewer's eye 90 as the probing light 96. The probing light 96can have a wavelength that is invisible to human eye, for exampleinfrared. The probing light 96 can have a wavelength that is visible tohuman eye, but not affecting viewer's normal vision, which can be anyone or any combination of: (1) the probing light has a small beam sizethat is insensitive to human eye; (2) the probing light is projectedonto the blind spot of the viewer's retina; (3) the probing light is thein the form of short pulses with pulse duration being too small for eye90 to sense.

Detector 94 detects the reflected probing light (Reflection light) 97from the retina 92. Detector 94 and transmitter 95 are composed ofelectric and/or optical circuitry. The transmitter 95 can be in the formof a transmitter array or a transmitter matrix. The detector 94 can bein the form of a detector array or a detector matrix.

The probing light 96 from the transmitter can be scanning inone-dimensional or two-dimensional patterns into the eye 90.

Reflection light 97 received by detector 94 is used to calculate theeye-information defined as any of, but not limited to, viewer's eye-lensfocus depth, eye-lens shape, eye-lens curvature, eyeball rotationalposition. Calculation of the said eye-information can be combined withthe probing light 96 information from transmitter 95. Reflection light97 can be monitored by the detector 94 for any of, but not limited to,intensity, angle, reflection spot position on retina 92, color, patternand beam shape, pattern shift, optical interference with the incomingprobing light 96. The calculated eye-information can be transmitted toanother device or temporarily stored in a data storage component notshown in FIG. 9.

Probing light 96 generation by the transmitter 95 can be integrated witha shutter function of the see-through substrate 93, for example instereo vision with active 3D glass where during the interval that theoutside image was temporarily shielded from the viewer's eye, theeye-information can be retrieved by enabling the probing light forminimal disturbance of normal viewing. Probing light 96 source can beany of, laser, LED, lamp, and can have an MEMS based mirror and/or lightscanning system. The see-through substrate 93 position is substantiallyfixed relative to the position of eye 90.

An image capturing device, for example a camera, can be integrated withthe see-through substrate 93 or be in proximity to the see-throughsubstrate 93, and moves together with the see-through substrate 93 tocapture the image that the viewer sees for comparison with thereflection light 97 information and calculate eye-information.

FIG. 10 is a schematic diagram illustrating the second implementationfor the step of sensing the re-focus intention of viewer according tothe embodiments of the current invention. This implementation is forretrieving the viewer's eye-information by eye-lens 101 reflectedprobing light 109. Every other aspect of this implementation is the sameas in the first implementation (FIG. 9) with the exception of following:the detector 104 and the detector 105 do not capture the reflectionlight of the probing light from the retina 102; detector 104 capturesreflection light 107 reflected from the front outside surface of thelens 101 when the probing light 109 enters the lens; detector 105captures reflection light 108 reflected from the back inside surface ofthe lens 101 when the probing light 109 exits the lens 101 and entersthe vitreous humor of the eye 100; either one or both of the reflectionlight 107 and 108 information received by the detector 104 and detector105 can be used to extract the eye-information; detector 104 anddetector 105 can be in the form of a detector array or a detectormatrix.

Optical interference pattern produced between any of the probing light109, reflection light 108, and reflection light 107 may be used toretrieve the re-focus intention of the viewer. Note that detector 104and detector 105 can be used together or only one of the two can be usedto retrieve the eye-information

FIG. 11A is a schematic diagram illustrating the third implementationfor the step of sensing the re-focus intention of viewer according tothe embodiments of current invention. This implementation is forretrieving the viewer's eye-information by projected pattern 114 onretina. Image screen 111 is the place where object image 119 (no shownin FIG. 11A) that viewer is viewing is being created, where the image119 is also adjusted in real-time following viewer's re-focus intent toprovide a re-focusable vision for the viewer eye 110.

Pattern 112 on the image screen 111 is used to help retrieve theviewer's eye-information. The pattern 112 can be produced by a light atwavelength that is invisible to human eye, for example infrared. Thepattern 112 can also be produced by a light at wavelength that isvisible to human eye, but not affecting viewer's normal vision. Thepattern 112 can be produced in very short time pulsed interval that isinsensitive to human eye 110. The pattern 112 can be producedoverlapping other image 119 shown to the viewer on the image screen 111.The pattern 112 can be produced interlacing with the image 119 shown tothe viewer with a shuttered mechanism, where pattern 112 is not shown atthe same time as the image 119, and pattern 112 showing time iscomparatively much shorter than the image 119. The pattern 112 can bevarying density, arrangement, shape, size and position over time to helpenhance the extraction of the eye-information. The pattern 112 positionon the image screen 111 can have a one dimensional or two-dimensionaltemporal oscillation.

See-through substrate 116 provides a supporting frame for the detector117 and allow viewer to see through. Light 113 from image screendisplayed to the viewer can pass through see-through substrate and formsoptical projection on the retina for viewer to see. See-throughsubstrate 116 can serve as part of the stereoscopic vision system thathelps images taken from same scene at different viewing angles beingshown each eye 110 of the viewer separately, so that viewer has astereoscopic vision impression. See-through substrate 116 with detector117 does not affect viewer's normal vision of the image on the screen111.

Pattern 112 produces projected pattern 114 image on the retina of theviewer. Pattern image 114 is further reflected by the retina and thedetector 117 receives the reflected pattern image 118 from the retina.The detector 117 can be in the form of a detector array or a detectormatrix. Reflection light 115 of the pattern 114 received by detector 117is used to calculate the eye-Information. Calculation of the saideye-information can be combined with the information of the pattern 112on the image screen 111. Reflection light 115 of the pattern 114 can bemonitored by the detector 117 for any of, but not limited to, intensity,position on retina, color, shape, size, density, arrangement, position,oscillation pattern and oscillation frequency. The calculatedeye-information can be transmitted to another device or temporarilystored in a data storage component that is not shown in FIG. 11A. Animage capturing device, for example a camera, can be integrated with thesee-through substrate 116 or be in proximity to the see-throughsubstrate 116, and moves together with the viewer's eye and see-throughsubstrate to capture the pattern 112 that is shown to the viewer on theimage screen for comparison with the reflection pattern 114 informationand calculate eye-information. The detector 117 is composed of electricand optical components.

FIG. 11B is a schematic diagram illustrating the fourth implementationfor the step of sensing the re-focus intention of viewer according tothe embodiments of current invention. This implementation is forretrieving the viewer's eye-information by reflected images from thelens.

Every other aspect for this implementation is the same as in the thirdimplementation (FIG. 11A) with the exception of following: the detectordoes not capture the reflection of the pattern on image screen from theretina; at least one detector captures reflection image 1124 of thepattern 1122 reflected from the front outside surface of the lens whenthe light from the pattern 1122 on screen 1121 enters the lens; at leastone second detector captures reflection image 1123 of the pattern 1122reflected from the back inside surface of the lens when the light fromthe pattern on screen exits the lens and enters the vitreous humor ofthe eye 110; either one or both of the reflection pattern 1123 and 1124information received by the first detector and second detector can beused to extract the eye-information; both first and second detectors1127 can be in the form of an array or a matrix; the first and seconddetectors 1127 can be same detector; during usage, it is possible to useonly one of the first and second detectors; during usage, it is possibleto use both of the first and second detectors; and the opticalinterference of the pattern 1123 and pattern 1124 can also be used tocalculate eye-information.

FIG. 12A is a schematic diagram illustrating the fifth implementationfor the step of sensing the re-focus intention of viewer according tothe embodiments of current invention. This implementation is forretrieving the viewer's re-focus intention by using a scanning probinglight beam 1241 across viewer's eyeball and simultaneously monitoringthe reflected optical signal from the eye 120.

When viewer's intention of re-focus happens, the eye-lens 121 of theviewer eye 120 can change in shape and curvature. The change of eye-lens121 shape in the form of compression or stretching 122 in the directionof the viewers' eye-sight causes the part of the eye 120 in front of theeye-lens to deform correspondingly. Such process is also called“accommodation” during an eye 120 re-focus process. The corneal shape123 of the eye 120 can also be deformed in small amount by the shape andcurvature change 122 of the eye-lens 121. In FIG. 12A, an opticalemitter 124 is used to project an directional probing light 1241 uponthe cornea and an optical detector 125 is used to detect the reflectionlight 1251 from the cornea. At different corneal shape 123 caused by thedifferent eye-lens shape change 122, the reflection light 1251 asreceived by the optical detector 125 also changes its intensity orreflection angle. By scanning the probing light 1241 across the viewer'seye 120, and by monitoring the reflection light 1251 change during thescan, the information of the eye-lens change as well as the pupilposition of the eye-ball can be measured.

FIG. 12B is a schematic diagram illustrating the scanning procedure ofthe probing light 1241 by oscillating the optical emitter 124 of FIG.12A. The optical emitter 124 oscillates from left to right in FIG. 12Band produces scanning of probing light 1241 across the eye 120 of theviewer. With the optical emitter also changes its rotational orientationin the direction normal to the scan direction, multiple discrete scanlines, 1291, 1292, 1293, 1294 and 1295 can be produced from top tobottom of the eye 120 with the scan lines covering the exposed eyeballarea between the eye lips 128. The optical detector 125 detects thereflection light 1251 from the viewer's eyes while probing light 1241scans across the eye 120.

The eye-information detection scheme as shown in FIG. 12A and FIG. 12Bhas the advantage over prior arts in the aspect of simpler structure andlower cost. The major components of this new scheme are the opticalemitter 124 and the optical detector 125. With the probing light 1241scanning positions well controlled and calibrated, mapping of the eye120 by the reflection light 1251 captured by the optical detector 125can be realized by simple electronics with low cost. The optical emitter124 can be low cost light emission diode (LED) or laser diode with goodlight directionality. The optical detector 125 can also be low costphotodiode with proper optical filter. This new scheme can also beintegrated into head-mounted supporting structures, for example in theform of eye-glasses, due to no complicated optics is required. Priorarts that detect pupil 126 positions, i.e. eye-tracking, generally useimaging of the user's eyes, which not only require complicated andexpensive optical lens system but also require sophisticated andexpensive electronics for image processing to retrieve pupil positioninformation.

The eye-information detection scheme as shown in FIG. 12A and FIG. 12Balso has the advantage over prior arts in the aspect of accurate focusdepth extrapolation, high precision and less interference fromenvironment. Spatial resolution of this new scheme is defined by thelight beam size and the scanning resolution of the probing light 1241,which can realize high precision with commercially available low costLED, laser diode and MEMS technologies, with spatial resolution reachingmicron level or smaller. For prior art image capture methods, such highresolution is either not economically achievable or having to useexpensive optical and electrical components. Additionally, the probinglight 1241 can also be modulated with single-tone high frequency patternthat enables lock-in technique detection of the reflection light, or itcan be modulated with digital patterns that enables high speed digitalfiltering, both of which can increase signal-to-noise-ratio (SNR) of themethod and is well beyond prior art method.

The eye-information detection scheme as shown in FIG. 12A and FIG. 12Bfurther has the advantage over prior arts in the aspect of simultaneousdetection of eye-lens change and pupil position change. In prior artschemes, due to the spatial resolution limitation and long detectiondistance of the optical system, it is only possible to detect the pupil126 position, i.e. eye-tracking. The new scheme of this invention asshown in FIG. 12A through FIG. 12D, with its ability to be integrated tohead-mount structure and the close proximity of both the optical emitter124 and optical detector 125 to the viewer's eye 120, the optical signalcaptured by the detector during the scanning of the probing light notonly can identify the position of the pupil, but also can be used toextract the information of the eye-lens change, which givesunprecedented advantage in faster focus depth calculation, lowercalculation complexity, and higher calculation accuracy.

Method of FIG. 12A and FIG. 12B can have any one or a combination ofbelow features: (1) there can be multiple optical detectors 125 tocapture the reflection light 1251 signal from the same probing light atdifferent locations relative to the eye 120; (2) there can be multipleoptical emitters 124 with each emitter 124 producing scan lines notexactly overlapping any of the scan lines produced by any other emitter124; (3) a single scan line of probing light 1241 can be in anydirection across the eye 120; (4) when probing light 1241 scans over thepupil area, reflection light 1251 from the eye 120 can be reflected fromany of: cornea, eye-lens 121 front surface facing cornea, eye-lens 121back surface facing retina; (5) probing light 1241 can be invisiblelight, and preferably infra-red light; (6) probing light 1241 can bevisible light but with intensity insensible by human eye; (7) probinglight 1241 can be modulated into pulsed patterns wherein the duty cyclesof the pulses are short enough such that the effective probing lightintensity is insensible by human eye; (8) probing light 1241 can bemodulated into pulsed patterns that has a single tone frequency whereinthe optical signal captured by the optical detector 125 also shows samesingle tone frequency pulse pattern, which can then be processed by anlock-in method that enhances the SNR of the detection result; (9)probing light 1241 can be modulated into pulsed patterns that representsa digital sequence wherein the optical signal captured by the opticaldetector 125 also shows same digital sequence pattern, which can then beprocessed by a digital filter that enhances the SNR of the detectionresult; (10) special pulsing patterns of the probing light 1241 canexist at the beginning, or at the end, or in the middle section of anyof the scan line 1291,1292,1293,1294,1295, to designate the beginning,ending, or within-scan locations of the scan. Such special pulsepatterns can be also used to identify the order of the different scanlines for spatial alignment of different scan lines during signalprocessing of the optical signal captured by the optical detector 125;(11) the optical emitter 124 oscillatory motion can be generated by adriving mechanism that can be based on any of: MEMS, magnetic force,piezoelectric effect, electrostatic effect, capacitive effect, orthermally induced shape change.

FIG. 12C and FIG. 12D are schematic diagrams illustrating the sixthimplementation for the step of sensing the re-focus intention of vieweraccording to the embodiments of current invention. All other aspects ofFIG. 12C and FIG. 12D are identical to FIG. 12A and FIG. 12B case,except that the scanning of the probing light 1241 is produced by areflection mirror or prism 1242, wherein the optical emitter 124 isstationary.

Method of FIG. 12C and FIG. 12D can have any one or any combination ofbelow features: (1) a single mirror or prism 1242 can be used to scanthe probing light 1241 in spatially discrete scan lines as shown in FIG.12D; (2) a series of mirrors or prisms 1242 can be used with single ormultiple optical emitters 124 with each mirror or prism 1242 producingone or more scan lines not exactly overlapping any one of the scan linesproduced by any other mirror or prism 1242; (3) an array of mirrors orprisms 1242 can be used with single or multiple optical emitters 124with each mirror or prism 1242 producing a light spot on the eye 120 andarea around the eye 120. By enabling the mirrors or prisms 1242 toproduce the light spots in a sequential order, effective scan lines canbe produced; (4) the mirror 1242 can be a mirror array that is the sameone being used to directly project image upon the retina of the viewer'seye 120 as described in FIG. 34, wherein scanning of the probing lightand image projection by the same mirror can be interlaced or multiplexedwith the same mirror or mirror arrays; (5) the oscillatory motion of themirror or prism can be generated by a driving mechanism that can bebased on any of: MEMS, magnetic force, piezoelectric effect,electrostatic force, capacitive force, or thermally induced shapechange.

FIG. 12E shows examples of the optical signal sensed by the opticaldetector 125 during scanning of the probing light 1241 in FIG. 12B andFIG. 12D. The X axes of all sub-figures in FIG. 12E are the physicalposition along each scan line across the eye 120, while the Y axes arethe strength of the optical signal sensed by an optical detector 125.The signal traces 12911, 12921, 12931, 12941 and 12951 are respectivelycorresponding to the scan traces of 1291, 1292, 1293, 1294 and 1295 ofFIG. 12B and FIG. 12D. For the examples of FIG. 12E, the probing lightis assumed to be infra-red light. The infra-red light reflects from theeye-ball area is stronger than from eye-lip and pupil. Pupil area corneareflects infra-red light the weakest due to highest absorption ofinfra-red light. Trace 12911 and trace 12951 both have two levels in thesignal strength, with the higher level in the center corresponding tothe probing light scanning over the eye-ball and lower level at thesides corresponding to the eye-lip. Traces 12921, 12931 and 12941 arefrom scans that pass across pupil, therefore they show lower signallevel at regions around the middle of the traces, with trace 12931having the highest downwards peak 12932 at the trace center.

The width 12933 and amplitude 12934 of the downwards peak 12932 can beused to calculate the position of the pupil and the lens changeinformation. With pupil position change, the horizontal position of thehighest amplitude point 12934 of the downwards peak 12932 can shift inthe trace 12931. Additionally, with pupil position change, the tracethat exhibits the largest downwards peak may also shift from 12931 toanother trace. The shift of the maximum downwards peak position betweentraces and along scan direction can be used to calculate pupil position.When eye-lens focus depth changes, shape change of the eye-lens cancause shape change of the cornea as shown in FIG. 12A and FIG. 12C. Withthe light reflecting from cornea, or from the eye-lens, or both, theshape change of cornea or eye-lens can produce a reflection lightchange, in intensity or in reflection angle or both, most likely at theboundary of the pupil. Such change will affect the pulse width 12933 orpulse shape of peak 12932. Thus, with capturing the reflection light1251 during scanning of probing light 1241, and with monitoring the peak12932 position, peak height 12934, peak width 12933, or pulse shape,information of the pupil position and eye-lens change can be retrievedwith signal processing and calculation.

It needs to be noted that although infra-red light and its lowerreflection by pupil is used as example in FIG. 12E, other lightwavelength with other reflection properties can also be used withoutlimitation. Multiple wavelength probing light can also be used at sametime. Additionally, scan lines as shown in FIG. 12E can be obtained frommore than one optical detectors 125 around the eye 120, for bettersignal capture and higher accuracy in calculation of eye-information.

For the scan lines 1291,1292,1293,1294,1295 of probing light 1241 shownin FIG. 12B and FIG. 12D, although straight parallel scan lines are usedas example, the scan lines are not limited to straight line or parallelslines. The scan lines can be any one or any combination of the belowtypes to efficiently cover the area of the viewer's eye: (1) at leasttwo sets of parallel straight or curved lines that cross each other atvarious crossing points with crossing angles between 0 to 90 degrees;(2) concentric circles; (3) at least two circles partially overlappingeach other; (4) at least two close-loop shapes overlapping each other;(5) one or more continuous scan lines with irregular scan traces thatcovers sufficient amount of the viewer's eye area; (6) a rotatingregular or irregular scan pattern; (7) at least one set of parallelstraight or curved lines; (8) at least two close-loop shapes with oneenclosed entirely by the other one.

FIG. 12F is a schematic diagram illustrating the pupil positions of theeyes of the viewer when viewer is focusing on a far point 1210, and FIG.12G is a schematic diagram illustrating the pupil positions of the eyesof the viewer when viewer is focusing on a near point 1220. In FIG. 12Fand FIG. 12G, although viewer is focusing on different points in spacethat are at different distances from the viewer, the right eye 1201pupil 1261 position and eye-sight direction 12031 is the same.Therefore, by only monitoring the pupil position, similar as in“eye-tracking” techniques used in prior arts, both eyes 1201 and 1202must be monitored at the same time to extrapolate the focusing point ofthe viewer's eye sight with extending the eye-sight line 12031 and 12041directions of both eyes 1201 and 1202, to find out the eye-sightcrossing points as the focus points. The prior art “eye-tracking”method, although is straight forward, requires tracking of both eyes andability to find the actual focus point with complicated electronics andalgorithm, which are slow in speed, expensive in implementation andinapplicable to viewers with disability in one of the two eyes.

With the ability to obtain eye focus depth information from eye-lenschange, monitoring both eyes is then not required. FIG. 12H shows anexample of utilizing FIG. 12A and FIG. 12B implementation method foridentifying the focus point of the viewer 1200 with monitoring a singleeye, i.e right eye 1201. With the ability of detecting right eye 1201eye-lens change and its focus depth, the focusing point of the viewercan be found locating on a focus circle 12052 with a radius of 12051with the focus circle 12052 centered on the viewer. The radius 12051 isdefined as the distance from the viewer that the viewer is focusing onby the eye 1201's detected focus depth. Then with the ability to detectthe position of the pupil, the eye sight 12031 direction of the righteye 1201 can be extrapolated. The crossing point of right eye sight12031 and the focus circle 12052 is then the focusing point 12053 of theviewer. Since when viewer focuses on a spatial point, both eyes willfocus at that same point, with locating the point of the focus for righteye 1201, it is also the point of focus of left eye.

To apply FIG. 12H scheme in applications as shown in FIG. 18 throughFIG. 33, the information of the exact location of focus point 12053 andeye sight 12031 direction are not required in certain embodiments. Withobtaining the focus circle 12052 and radius 12051 from the eye-lenschange, of all objects that are being shown to the viewer, the ones thatare on or in close proximity to the focus circle 12052, can be broughtinto clear focus to the viewer's eye 1201, and then allow the viewer eyeto select and focus on the object of interest on the focus circle byviewer's choice, i.e. finding and looking at the object of interest. Inthis way, the complexity of eye information detection is further reducedwith only requiring detection of focus depth change information, andeye-information processing speed is faster and cost of implementation isalso cheaper.

FIG. 13A is a schematic illustrating the fifth implementation for thestep of sensing the re-focus intention of viewer according to theembodiments of current invention with a contact-lens type of see-throughsubstrate 131, which is in direct contact with the eye ball andsubstantially covers the pupil of the eye 130, being used for the samepurpose of the see-through substrates described in earlier figures.Instead of a see-through substrate that is positioned apart from theviewer's eye with a gap, a contact-lens type of see-through substrate131 can be used for fulfilling the functions of the see-throughsubstrates 93, 103, 116 and 1126 as respectively illustrated in FIG. 9,FIG. 10, FIG. 11A and FIG. 11B.

FIG. 13B is a schematic illustrating a contact-lens type of see-throughsubstrate 131 (“contact lens”) that is in direct contact with the eyeball and substantially covers the pupil of the eye 130, with embeddedelectronics 132, optical emitter 133 and optical detectors 134 torealize FIG. 9 and FIG. 10 type of functions to detect focus depthchange of the eye 130. Electronics 132 is preferred located close to theouter edge of the contact lens to avoid interfering with the viewing ofthe eye 130 through pupil 136. One obvious advantage of FIG. 13B type ofsolution over prior arts is that the optical emitter 133 and opticaldetector 134 are always moving together with the pupil position. Duringmovement of eye ball 130, relative position of the substrate 131together with all embedded components to the pupil is fixed. Thus, thedetection accuracy is greatly enhanced. Electronics 132 is providingpower to and communicating with optical emitter 133 and optical detector134. Electronics 132 can also have components interacting wirelesslythrough electromagnetic coupling to an external circuitry not shown inFIG. 13B to realize functions of: (1) harvesting external powerwirelessly; (2) transmitting data into electronics 132 to controlemitter 133 or transmitting data of optical signal detected by detector134 out from electronics 132. Optical emitter 133 is located in closeproximity, and preferably directly above, the pupil 136. Optical emitter133 produces optical radiation towards inside the eye 130 through pupil136 with a pre-determined optical pattern. Such optical pattern can becontinuous light beam with constant intensity, light pulses, orcontinuous light team with varying intensity over time. At least oneoptical detector 134 exists in substrate 131. Optical detector 134 candetect any one, or any combination, of following properties: (1)reflected light intensity change over time at a specific location withinthe substrate 131; (2) reflected light intensity at various locationswithin the substrate 131; (3) time delay between different reflectedpulses at a specific location with the substrate 131; and (4) time delaybetween different reflected pulses at various locations within thesubstrate 131. With the detected light signal from detector 131 alone,or in combination with emitted light signal from emitter 133, the focusdepth information of the eye 130 can be retrieved. An external circuitrynot shown in FIG. 13B can be used to monitor the electronics 132 spatialposition change following the rotation of the eye 130, such that boththe direction of eye sight and the focus depth can be obtained tore-produce exact focus point in space by the eye 130, whereinelectromagnetic coupling between at least one component in the externalcircuitry and at least another component in electronics 132 is used forsuch monitoring.

There can be more than one optical emitter 133 and more than one opticaldetector 134 embedded in the substrate 131. The optical emitter 133 canemit visible light or infra-red light. When optical emitter 133 oroptical detector 134 are in close proximity to the pupil, or directlyabove pupil, to avoid interfering with vision of eye 130, the emitter133 or detector 134 can be made transparent, or can be in the size smallenough that will not affect vision, for example in the size smaller than100 micrometers.

The optical emitter 133 or optical detector 134 can be also be part ofthe electronics 132 and located away from the pupil 136 same as theelectronics 132. In this case, optical paths connect the output of theemitter 133 or input of the detector 134 towards the location of thepupil 136, and reflective components, for example micro-mirrors,terminate at the other ends of the optical paths at the locations of 133and 134 shown in FIG. 13B to emit light into the pupil 136 or collectlight reflected back from the pupil 136. The light paths and reflectivecomponents are both small enough to avoid affecting eye 130 vision, forexample with maximum width less than 100 micrometers.

One example of operation of FIG. 13B scheme is that emitter 133 emitslight beam into pupil 136. The various surfaces of cornea, eye-lens, andretina reflect and scatter the incident light from the emitter 133. Whenthe reflected light reaches detector 134, it produces a light pattern ofdispersion. With various eye length focusing depth, such dispersionpattern changes either or both of its size and its shape. By correlatingthe dispersion pattern change with intended focus depth, the intendedfocus depth of the eye 130 can be extrapolated.

Another example of operation of FIG. 13B scheme is that emitter 133produces light pulses into pupil 136. The various surfaces of cornea,eye-lens, and retina reflect the incident light pulses at different timewhen the incident light passes through its optical path into the eye 130until reaching the retina layer. When reflected light pulses fromdifferent surfaces passes through various eye components, for example,eye-lens, cornea, and are diffracted variously before reaching thedetector 134 and the detector detects the reflect pulses arriving time.From the time delay between two or more reflected light pulses thatreach detector 134, the intended focus depth of the eye 130 can becalculated.

Still another example of operation of FIG. 13B scheme is that emitter133 produces light beam into pupil 136 with a given incident angle tothe surface of the eye-lens. The light beam is then reflected from thesurfaces of the eye-lens when light beam passes through the eye-lens andproduces at least one reflection light point on the substrate 131 whichis then detected by at least one of the detectors 134. For lightreflected from eye-lens insider surface, it is also refracted by theeye-lens during the reflection. When eye focus depth changes due toeye-lens shape change, the reflected light is reflected into differentdirections due to surface curvature change of the eye-lens and thus thereflection light point on the substrate moves to a different location onsubstrate 131. By correlating the position of the reflection lightpoints with the intended focus depth, the intended focus depth of theeye 130 can be extrapolated. The circuitry 132 may contain any of or anycombination of, but not limited to, metal circuit, organic circuit,optical circuits, MEMS sensor, piezo sensor, capacitance sensor,magnetoelastic sensor, pressure sensor, deformation sensor, RF circuit.

FIG. 13C is a schematic illustrating a focus-depth detection with usingsame as FIG. 13B scheme with addition of a fixed frame 135 in closeproximity to the eye 130. The fixed frame 135 can serve the one or bothfunctions of: (1) providing power to the electronics 132 wirelesslythrough electromagnetic coupling to electronics 132, for example byinductive coupling or wireless antenna; (2) detecting the spatialposition change of the pupil 136 through monitoring the spatial positionchange of the electronics 132 relative to the frame 135, such that boththe direction of eye sight and the focus depth can be obtained tore-produce exact focus point in space of the eye, whereinelectromagnetic coupling between at least one component in the frame 135and at least another component in electronics 132 is used for suchmonitoring.

FIG. 14 is a schematic diagram illustrating the eighth implementationfor the step of sensing the re-focus intention of viewer according tothe embodiments of current invention. This implementation is forretrieving the viewer's eye-information by electrical method.

Contact-lens 142, which is in direct contact with the eye ball andsubstantially covers the cornea of the eye 130, provides a supportingframe for the circuitry 146 embedded in the contact-lens 142. Light fromimage and scene that are displayed to the viewer can pass through thecontact-lens 142 and allows viewer to see through. Contact-lens 142 canserve as part of the stereoscopic vision system that helps images takenfrom same scene at different viewing angles being shown to each eye ofthe viewer separately, so that viewer has a stereoscopic visionimpression. Contact-lens 142 with embedded circuitry 146 does not affectviewer's normal vision of the shown images or scene.

When viewer's intention of re-focus happens, the eye-lens 141 of theviewer can change in shape and curvature. The change of eye-lens 141shape in the form of compression or stretching 143 in the direction ofthe viewers' eye-sight causes the part of the eye in front of theeye-lens to deform correspondingly. The cornea 144 of the eye can bedeformed in small amount by the shape and curvature change 143 of theeye-lens 141, and exerts different forces 144 onto the contact-lens 142.

The circuitry 146 embedded in the contact-lens can be used to sense thedeformation of the of the contact-lens 142, or pressure and stretchforce 145 change exerted on the contact lens 142. The circuitry 146 maycontain any of or any combination of, but not limited to, metal circuit,organic circuit, optical circuits, MEMS sensor, piezo sensor,capacitance sensor, magnetoelastic sensor, pressure sensor, deformationsensor, RF circuit. The circuitry 146 may be powered by any of, but notlimited to, an optical to electrical power converter, an electricalpower source, an RF power detector, body temperature of viewer, chemicalreaction within the contact-lens by moisture of the eye, an embeddedbattery in the contact-lens, eye-lips closing & opening mechanicalforces, wireless electromagnetic coupling to external power source. Thecontact-lens 142 can be operating together with an external see-throughsubstrate put in front of the eye to achieve re-focus sensing andstereoscopic vision.

FIG. 15 is a schematic diagram illustrating the ninth implementation forthe step of sensing the re-focus intention of viewer according to theembodiments of current invention. This implementation is for retrievingthe viewer's eye-information by brain wave pattern. When an object 154is projected into the eye and forms image 155 on the retina of the eye,the eye nerves 152 sense the image information and transmits suchinformation to brain 151 through neural pathways 156. Brain-wave patternassociated with vision and intention of vision will be generated afterimage information perceived by brain 151. Such brain-wave patterns ofre-focus can be pre-characterized or pre-trained for the viewer.Brain-wave sensors 153 are attached to the skull of viewer. In certainmedical applications, brain-wave sensors 153 can be in contact with thebrain cells inside the skull for better brain-wave capturing. Whenbrain-pattern changes, it is compared with a database of knownbrain-patterns and their intended actions. If a brain-pattern ofre-focus and re-focus direction retrieved from database, or generatedwith data from the database, can be matched to the brain-patterncaptured, a re-focus event is generated.

Now, coming back to the First Embodiment. For the Step 103 ofcalculating the desired focus depth (Step 103) and retrieving image(Step 104), a computing system is used to obtain information of viewer'seye-lens, eyeball or brainwave pattern change from the sensors sensingsuch information from the viewer, and calculate the desired re-focusdepth of the viewer about the image currently shown to the viewer. Forbrain-pattern recognition of viewer's re-focus intention, a brain-wavepattern database also provides information to the computing system tocompare to the received brain pattern. Calculation of intended focusdepth in Step 103 can be computed by the eye-lens, or together witheye-ball change, information obtained in Step 102, and optionallytogether with the image currently being displayed to the viewer. Animage capturing device, for example a camera, can be in close proximityto the viewer's eyes to capture the scene that the viewer is currentlybeing exposed to, wherein the eye-lens focus depth, or together with eyepupil position, can be compared to the captured image to calculate theobject of interest that is being focused upon. The display device wherethe image is displayed can also provide the current image informationdirectly to the computing system. After a desired re-focus depth iscalculated, for single viewer case, the computing system then retrievesthe correct image with the desired focus depth from the recording mediaor recorded image database and display such image on the display device.For the Step 105 of displaying the retrieved image, if the displaydevice displays single focus depth image only, only single viewer isallowed. To share the same display between multiple viewers,multiplexing device is now required.

For the Step 105 of displaying images on the same display for multipleviewers with different intended focus-depth, multiplexing device is nowrequired. For multiple viewer case, the first option is that the imagesof same scene but with different focus-depths are multiplexed bytime-slot to be displayed on the same display and selectively shown by ashuttered image multiplexing device to the viewer with matching intendedfocus-depth. The see-through substrates that viewers view through can bean image multiplexing device, to differentiate the different focus deptheach viewer desires, so that different viewer may see same displayedscene with different focus depth into the same scene. An example of themultiplexing is that images of same scene but with different focus-depthare sequentially shown on the same display to a group of viewers. Theviewers with different intended focus-depth through changing theireye-lens can each only view one of the different sequentially displayedimages due to the shutter function of the see-through substrates each ofthe viewer view through, where the see-through substrates synchronizewith the display regarding the sequence of sequentially displayeddifferent focus-depth images and only allow the image that has correctfocus-depth to be displayed to the viewer that has same intendedfocus-depth. Other images with other focus-depths that are not matchingthe intended focus-depth of the viewer are blocked by the shutter of thesee-through substrate so that the viewer cannot see. In this way, eachviewer always sees a scene or a changing scene that is always with thecorrect focus-depth according to the viewer's own intended focus depth.

FIG. 16 is a schematic diagram illustrating the second option for theStep 105 of displaying the retrieved image to multiple viewers. Multipleviewers may share the same display where each viewer has a dedicateddisplay unit of each pixel. Multiplexing device is required to sharesame display between multiple viewers. Each viewer has own retrievedimage to be displayed on the same screen. Each viewer can only viewassigned area of the screen. For the four adjacent pixels 161, 162, 163and 164 shown, each viewer can only view one area within each pixel asassigned to each viewer: Viewer 1 sees four white color areas at theupper left corner of each pixel, which produce effective white color;Viewer 2 sees two white and two black color areas at the upper rightcorner of each pixel, which produce effective gray color; Viewer 3 seesone white and three black areas at the lower left corner of each pixel,which produce effective dark gray color; Viewer 4 sees four black areasat the lower right corner of each pixel, which produce effective blackcolor. Such multiplexing can be achieved by synchronized shuttering ofimage by a shuttered device with-in the see-through substrate thatviewers view through, where the shuttered device synchronized with thedisplay of pixels to each viewer.

FIG. 17 is a schematic diagram illustrating the third option for thestep of displaying the retrieved image according to the embodiments ofcurrent invention. Multiple viewers may share a same display with thedisplay showing multiple focus depth images of the same scene.Multiplexing device is required to share same display between multipleviewers. Images for various focus depth of the eye are displayedsimultaneously on the same screen. Each pixel on the display containsmultiple areas with each area dedicated to a different focus depth. Eachviewer can only see the areas with the same focus depth within allpixels at any instant time. Each viewer's desired focus depth is sent tothe multiplexing device within the see-through substrate that the viewersee through. Each viewer's multiplexing device is adjusted to thedesired focus depth and shifts between different areas of the pixelshaving different focus depth to achieve effective focus depth change.For the four adjacent pixels 171, 172, 173 and 174 shown in FIG. 17,each viewer can only view the areas with the same focus depth of allpixels at any instant time. If a viewer's desired focus depth is FocusDepth 1, the multiplexing device then allows only the areas marked inFIG. 17 as “Focus Depth 1” to be shown to the viewer. If the viewerwants to focus to Focus Depth 4, the multiplexing device then adjustsand allows only the areas marked “Focus Depth 4” to be shown to theviewer. Such multiplexing can be achieved by synchronized shuttering ofimage shown on screen and the multiplex device within the see-throughsubstrate.

FIG. 18 is a schematic flow diagram illustrating the first embodimentwherein eye focus-depth sensing for eye-information are used:(Step-1001) A scene 181 of objects is recorded by a recording device 183as in 182; (Step-1002) The recorded image 184 of the scene 181 is storedin a recording media or a database of recorded image 185; (Step-1003) Asensor 189 is positioned in proximity to the viewer's eye 180 anddetects re-focusing information from viewer's eye 180; (Step-1004) Thesaid re-focusing data is transmitted as in 1893 to the computing device1895 for computing desired focus depth of the viewer; (Step-1005) Thedevice 1895 computes the desired focus depth of the viewer anddetermines the image to request from 185 media or database that hasdesired focus depth of the viewer; (Step-1006) The device 1895 sendsrequest to 185 recording media or database to request image with desiredfocus depth of the viewer as in 1894; (Step-1007) The device 185 sendsrequested image with desired focus depth of the viewer to the imagedisplay device 188 as in 186; (Step-1008) The device 186 displays therequested image with desired focus depth to the viewer.

In Step-1005, the current image shown on the image display device 188may optionally be used as an input to the device 1895 to compute desiredfocus depth of the viewer as in 187. Optional glasses 1891 can beintegrated with sensor 189 and positioned in front of the viewer's eye180 to allow viewer to see through, where the glasses 1891 can have thefunctions to enable any of: stereo vision, multiplexing differentviewers to share same display, powering sensor 189, communicatingbetween sensor 189 and device 1895, storing eye information detected bysensor 189, or provide a fixed spatial reference for detector 189 todetect eye 180 pupil position. In the case of multiple users sharingsame display, in Step-1006, the device 1895 can send to glass 1891 ofeach viewer the desired focus depth information 1892 of the images shownon display 188 to enable different user seeing different focus depthimages on the same image display 188.

Typically, when eye 180 focus depth changes the viewer sees objects atdifferent spatial distances from the eye. Only displaying image on fixeddisplay 188 will not replicate this real-life function and re-focusablevision will not work because the eye 180 is focusing on spatialdistances from the eye 180 other than the place of the display 188. InStep-1008, an optical imaging system 18912 with a variable effectivefocus depth can be disposed in the glass 1891 that the viewer's eye 180sees through, wherein the effective focus depth of the optical system18912 real-time and automatically adjusted to the viewer's eye-lensfocus depth change according to the focus depth information 1892 sentfrom device 1895, such that the image shown on same display 188 withfixed distance to eye 180 can appear to the viewer to be at differentdistances from the viewer when eye 180 intended focus-depth changes, andthe images shown on the display 188 always appears focused on the retinaof the viewer's eye at various viewer's eye-lens focus depth. Suchoptical image system 18912 can be any of: a single optical lens withmechanical positioning, a series or an array of optical lenses withmechanical positioning, a variable focus depth optical component that iscomposed of electrically-controlled refractive index material, anoptical component whose effective optical path for light passing throughcan be changed by an electrical signal, and an optical component basedon micro-electro-mechanical-system (MEMS) actuated lens, mirror or prismarrays.

FIG. 19 is a schematic flow diagram illustrating the first embodimentwherein brain-wave pattern sensing of re-focus intention are used. Allother steps, descriptions and procedures are same as in FIG. 18 case,except the following steps: (Step-1003) Brain-wave sensor 190 ispositioned in contact with the viewer's head to sense the brain-wavepattern of the viewer; (Step-1004) The said brain-wave pattern istransmitted as in 1993 to the device 1995 for computing desired focusdepth of the viewer; (Step-1005) The device 1995 computes the desiredfocus depth of the viewer with an additional input from a brain-wavepattern data-base 199, and determines the image with correct focus-depthto request from 195 media or database that matches the desired focusdepth of the viewer.

The second embodiment of the current invention is also for static ormotion pictures. The method according to the second embodiment includesthe steps of: (Step 201) Having a recording media containing images ofthe same scene where images are recorded simultaneously with differentfocus depth into the same scene; (Step 202) Active sensing the re-focusintention of viewer by monitoring the physiological change of viewer'svision related body function including viewer's eye-lens change, withoutviewer's active participation or physical action, and generating suchphysiological change information; (Step 203) Calculating intended focusdepth or intended focused object in the scene from the physiologicalchange information from Step 202; (Step 204) Retrieving the images withintended focus depth from the recording media containing recorded imagesfrom Step 201; (Step 205) Display retrieved image from Step 204 to theviewer's eyes.

In Step 202, the said physiological change of viewer's vision relatedbody function can also include the rotational position of the viewer'seye pupil.

In Step 205, an optical imaging system 18912 with a variable effectivefocus depth can be disposed in the optical path between the image andthe viewer's eye, where the effective focus depth of the system isautomatically adjusted to the viewer's eye-lens focus depth change inreal time according to the physiological change information from Step202, such that the image of Step 205 shown on the same screen appearsfocused on the retina of the viewer's eye at various viewer's eye-lensfocus depth. Such optical image system can be any of: a single opticallens with mechanical positioning, a series or an array of optical lenseswith mechanical positioning, a variable focus depth optical componentthat is composed of electrically-controlled refractive index material,an optical component whose effective optical path for light passingthrough can be changed by an electrical signal, and an optical componentcomposed of micro-electro-mechanical-system (MEMS) actuated lens, mirroror prism arrays that performs effectively as an optical lens or anoptical concave or convex mirror.

All other aspects in the second embodiment are identical to those in thefirst embodiment expect that Step 101 method of simultaneously recordingimages of the same scene with different focus depth on recording mediaare not specified. Actual method to record images with various focusdepth is not limited to the methods as described in the firstembodiment. The second embodiment focuses on the method to achievereal-time re-focus by measuring the viewer's re-focus intention andutilizing existing recorded images from the recording media. Steps 202,203, 204 and 205 in the second embodiment are same as Steps 102, 103,104 and 105 in the first embodiment.

FIG. 20 is a schematic flow diagram illustrating the second embodimentwherein eye-lens and eye-ball sensing of eye-information are used. Allother steps, descriptions and procedures are same as in FIG. 18 case,except Step-1001 and Step-1002 are removed, wherein 205 recording mediaor recorded image database already exists and contains imagessimultaneously recorded from the same scene with different focus depth.

FIG. 21 is a schematic flow diagram illustrating the second embodimentwherein brain-wave pattern sensing of re-focus intention are used. Allother steps, descriptions and procedures are same as in FIG. 19 case,except Step-1001 and Step-1002 are removed, wherein 215 recording mediaor recorded image database already exists and contains imagessimultaneously recorded from the same scene with different focus depth.

The third embodiment of the current invention is for enhanced humanvision. The method according to the third embodiment includes the stepsof: (Step 301) Having an image recording and transmission device thathas at least one adjustable component that changes the focus depth ofthe device during recording process of a scene; (Step 302) Activesensing the re-focus intention of viewer by monitoring the physiologicalchange of viewer's vision related body function including viewer'seye-lens change, without viewer's active participation or physicalaction, and generating such physiological change information; (Step 303)Calculating intended focus depth or intended focused object in the scenefrom the physiological change information from Step 302; (Step 304)Adjusting said adjustment component in Step 301 to reach intended focusdepth of said device in Step 301; (Step 305) Recording and transmittingimage by said device in Step 301 and displaying the transmitted image tothe viewer's eyes.

Compared to the first embodiment, when a desired focus depth iscalculated, instead of retrieving an image with the desired focus depthfrom the recording media or image database, the focus depth of therecording device into the scene is adjusted to the desired focus depthof the viewer. After recording a new image of a live scene with theadjusted focus depth, the newly recorded image with focus depth matchingviewer's desired focus depth is then displayed to the viewer as theresult of the viewer's intention to re-focus.

In Step 302, the said physiological change of viewer's vision relatedbody function can also include the rotational position of the viewer'seye pupil.

In Step 305, an optical imaging system 18912 with a variable effectivefocus depth can be disposed in the optical path between the image andthe viewer's eye, where the effective focus depth of the system isautomatically adjusted to the viewer's eye-lens focus depth change inreal time according to the physiological change information from Step302, such that the image of Step 305 showing on the same display appearsfocused on the retina of the viewer's eye at various viewer's eye-lensfocus depth. Such optical image system can be any of: a single opticallens with mechanical positioning, a series or an array of optical lenseswith mechanical positioning, a variable focus depth optical componentthat is composed of electrically-controlled refractive index material,an optical component whose effective optical path for light passingthrough can be changed by an electrical signal, and an optical componentcomposed of micro-electro-mechanical-system (MEMS) actuated lens, mirroror prism arrays that performs effectively as an optical lens or anoptical concave or convex mirror.

FIG. 22 is a schematic flow diagram illustrating the third embodimentwherein eye-lens and eye-ball sensing of eye-information are used,including: (Step-3001) A scene 221 of objects is recorded by a recordingdevice 223 having a focus depth adjustment component 224 as in 222;(Step-3002) A sensor 229 is positioned in proximity to the viewer's eye220 and collects the re-focusing information or data from viewer's eye220; (Step-3003) The said re-focusing data is transmitted as in 2293 tothe device 228 for computing desired focus depth of the viewer;(Step-3004) The device 228 computes the desired focus depth of theviewer and determines the adjustment needed in said focus depthadjustment component 224; (Step-3005) The device 228 sends request tofocus depth adjustment component 224 to adjust to desired focus depth ofthe viewer as in 2294; (Step-3006) The recording device 223 recordsimage of current scene 221 of objects with adjusted focus depthadjustment component 224 and the said recorded image is transmitted toimage display 226 as in 225; (Step-3007) The image display device 226displays the updated image sent from recording devices 223 with desiredfocus depth of the viewer.

In Step-3004, the current image shown on the image display device 226may optionally be used as an input to the device 228 to compute desiredfocus depth of the viewer as in 227. Optional glasses 2291 can beintegrated with sensor 229 and positioned in front of the viewer's eye220 to allow viewer to see through, where the glasses 2291 can have thefunctions to enable any of: stereo vision, multiplexing differentviewers to share same display 226 and control same focus depthadjustment component 224, powering sensor 229, communicating betweensensor 229 and device 228, storing eye information detected by sensor229, or providing a fixed spatial reference for detector 229 to detecteye 220 pupil position. In the case of multiple users sharing samedisplay, in Step-3007, the device 228 can send to glass 2291 of eachviewer the desired focus depth information 2292 of the images shown ondisplay 226 to enable different user seeing different focus depth imageson the same image display 226; also in Step-3005, the device 228 cansend request to focus depth adjustment component 224 as in 2294 toadjust to desired focus depths of all viewers which are implemented bythe component 224 in a sequential and time slotted manner, whereas oneviewer's desired focus depth is realized by the component 224 in anassigned time slot and image recorded by device 223 during that assignedtime frame will be only shown to the said viewer by display 226 with theuse of a multiplexing device in glass 2291.

In Step 3007, an optical imaging system 18912 with a variable effectivefocus depth can be disposed in the glass 2291 that the viewer's eye 220sees through, wherein the effective focus depth of the system real-timeand automatically adjusted to the viewer's eye-lens focus depth changeaccording to the focus depth information 2292 sent from device 228, suchthat the image shown on the same display 226 always appears focused onthe retina of the viewer's eye at various viewer's eye-lens focus depth.Such optical image system can be any of: a single optical lens withmechanical positioning, a series or an array of optical lenses withmechanical positioning, a variable focus depth optical component that iscomposed of electrically-controlled refractive index material, anoptical component whose effective optical path for light passing throughcan be changed by an electrical signal, and an optical componentcomposed based on micro-electro-mechanical-system (MEMS) actuated lens,mirror or prism arrays.

FIG. 23 is a schematic flow diagram illustrating the third embodimentwherein brain-wave pattern sensing of re-focus intention are used. Allother steps, descriptions and procedures are same as in FIG. 22, exceptfollowing steps: (Step-3002) Brain-wave sensor 230 is positioned incontact with the viewer's head to sense the brain-wave pattern of theviewer; (Step-3003) The said brain-wave pattern is transmitted as in2393 to the device 238 for computing desired focus depth of the viewer;(Step-3004) The device 238 computes the desired focus depth of theviewer with an additional input from a brain-wave pattern data-base 239,and determines the adjustment needed in the focus depth adjustmentcomponent 234.

The fourth embodiment of the current invention is for artificial realityor augmented reality. The method according to the fourth embodimentincludes the steps of: (Step 401) Having an artificial image generationdevice, for example a computer or an image processor, that has at leastone input parameter that controls the focus depth during imagegeneration process of a scene; (Step 402) Active sensing the re-focusintention of viewer by monitoring the physiological change of viewer'svision related body function including viewer's eye-lens change, withoutviewer's active participation or physical action, and generating suchphysiological change information; (Step 403) Calculating intended focusdepth and/or intended in-focus objects in the scene from thephysiological change information from Step 402; (Step 404) Adjusting theinput parameter in Step 401 to reach intended focus depth of the scenegenerated by the image generation device in Step 401; and (Step 405)Generating a scene by the generation device in Step 401 and displayingthe image of the generated scene to the viewer's eyes.

In Step 402, the said physiological change of viewer's vision relatedbody function can also include the rotational position of the viewer'seye pupil.

In Step 405, an optical imaging system 18912 with a variable effectivefocus depth can be disposed in the optical path between the image andthe viewer's eye, where the effective focus depth of the system isautomatically adjusted to the viewer's eye-lens focus depth change inreal time according to the physiological change information from Step402, such that the image of Step 405 displayed on the same screenappears focused on the retina of the viewer's eye at various viewer'seye-lens focus depth. Such optical image system can be any of: a singleoptical lens with mechanical positioning, a series or an array ofoptical lenses with mechanical positioning, a variable focus depthoptical component that is composed of electrically-controlled refractiveindex material, an optical component whose effective optical path forlight passing through can be changed by an electrical signal, and anoptical component composed of micro-electro-mechanical-system (MEMS)actuated lens, mirror or prism arrays that performs effectively as anoptical lens or an optical concave or convex mirror.

Compared to first embodiment, when a desired focus depth is calculated,instead of retrieving an image with the desired focus depth from therecording media or image database of as in first embodiment, in fourthembodiment, a parameter controlling the focus depth of the imagegenerated by the image generation device is adjusted and a new image isgenerated with the desired focus depth. The new image is then displayedto the viewer as the result of the viewer's intention to re-focus.

For Step 401, the image display can be an oblique display allowing imageto be shown to viewer by itself, or a transparent see-through displayallowing image to overlap a live scene that viewer sees.

FIG. 24 is a schematic flow diagram illustrating the fourth embodimentwherein eye-lens and eye-ball sensing of eye-information are used,including: (Step-4001) An image generation device 241 producinggenerated image 242 and having a focus depth adjustment parameter 243 asone input of the image generation process; (Step-4002) A sensor 249 ispositioned in proximity to the viewer's eye 240 and collects there-focusing information or data from viewer's eye 240; (Step-4003) Thesaid re-focusing data is transmitted as in 2493 to the device 248 forcomputing desired focus depth of the viewer; (Step-4004) The device 248computes the desired focus depth of the viewer and determines theadjustment needed of said focus depth adjustment parameter 243;(Step-4005) The device 248 sends request to image generation device 241to adjust focus depth adjustment parameter 243 according to the desiredfocus depth of the viewer as in 2494; (Step-4006) The image generationdevice 241 generates image 242 reflecting the desire focus depth of theeye 240 with adjusted focus depth adjustment parameter 243 and thegenerated image 242 is transmitted to image display device 245 as in244; (Step-4007) The image display device 245 displays the updated image242 sent from the image generation devices 241 to the viewer.

In Step-4004, the current image shown on the image display device 245may optionally be used as an input to the device 248 to compute desiredfocus depth of the viewer as in 247. Optional glasses 2491 can beintegrated with sensor 249 and positioned in front of the viewer's eye240 to allow viewer to see through, where the glasses 2491 can have thefunctions to enable any of: stereo vision, multiplexing differentviewers to share same display 226 and control same focus depthadjustment parameter 243, powering sensor 249, communicating betweensensor 249 and device 248, storing eye information detected by sensor249, or providing a fixed spatial reference for detector 249 to detecteye 240 pupil position. In the case of multiple users sharing samedisplay, in Step-4007, the device 248 can send to glass 2491 of eachviewer the desired focus depth information 2492 of the images shown ondisplay 245 to enable different user seeing different focus depth imageson the same image display 245; also in Step-4005, the device 248 cansend request to focus depth adjustment parameter 243 as in 2494 toadjust to desired focus depths of all viewers which are implemented bythe parameter 243 and device 241 to generate multiple images of 242 ofsame scene with each image reflecting one viewer's desired focus andsame image will only be shown to the said same viewer by display 245with the use of a multiplexing device in glass 2491.

In Step 4007, an optical imaging system 18912 with a variable effectivefocus depth can be disposed in the glass 2491 that the viewer's eye 240sees through, wherein the effective focus depth of the system real-timeand automatically adjusted to the viewer's eye-lens focus depth changeaccording to the focus depth information 2492 sent from device 248, suchthat the image shown on same display 245 always appears focused on theretina of the viewer's eye at various viewer's eye-lens focus depth.Such optical image system can be any of: a single optical lens withmechanical positioning, a series or an array of optical lenses withmechanical positioning, a variable focus depth optical component that iscomposed of electrically-controlled refractive index material, anoptical component whose effective optical path for light passing throughcan be changed by an electrical signal, and an optical componentcomposed based on micro-electro-mechanical-system (MEMS) actuated lens,mirror or prism arrays.

FIG. 25 is a schematic flow diagram illustrating the fourth embodimentwherein brain-wave pattern sensing of re-focus intention are used. Allother steps, descriptions and procedures are same as in FIG. 24 case,except following steps: (Step-4002) Brain-wave sensor 250 is positionedin contact with the viewer's head to sense the brain-wave pattern of theviewer; (Step-4003) The said brain-wave pattern is transmitted as in2593 to the device 258 for computing desired focus depth of the viewer;(Step-4004) The device 258 computes the desired focus depth of theviewer with an additional input from a brain-wave pattern data-base 259,and determines the adjustment needed in the focus depth adjustmentparameter 253.

In Step 101, Step 201, Step 301 and Step 401, the image recording, orthe recorded image, or the image recording device can be any of: (1)stereoscopic to achieve re-focusable stereo vision; and (2) conventionalnon-stereoscopic to achieve re-focusable plain vision.

In Step 102, Step 202, Step 302 and Step 402, the active sensing of there-focus intention of viewer can be any of: (1) by monitoring the changeof shape or curvature of any of: the viewer's eye-lens, cornea, andeyeball rotation by an optical method involving at least an opticalemitter and an optical detector; (2) by monitoring the change of theprojected image on the retina of the viewer's eye, where the projectedimage can be special patterns that are designed for sensing of re-focusintention, or the objects in the projected image that are focusedclearer than other objects in the image, where these said clearerobjects in the actual view that viewer is seeing are used to indicateviewer's focus depth and focusing point; (3) by monitoring the change ofshape or curvature of any of: the viewer's eye-lens, cornea, and eyeballrotation, by an electrical method without using optical emitter oroptical detector; and (4) by monitoring the brain wave pattern change ofthe viewer.

In Step 103, Step 203, Step 303 and Step 403, the calculation ofintended focus depth can be any of: (1) by the physiological changeinformation of viewer's eye; (2) by the image currently being displayedto the viewer together with the physiological change information. Animage capturing device, for example a camera, can be in close proximityto the viewer's eyes to capture and/or record the scene that the vieweris being exposed to, wherein the eye-ball position and/or eye-lens focusdepth can be compared to the captured image to calculate the objects ofinterest that need being focused upon.

In Step 304 of the third embodiment, the adjustable component can be (1)lens or lens array, mirror or mirror array, lens and mirror combination;or (2) mechanical or electrical mechanism that changes the focusingdepth of the said recording device.

In Step 404 of the fourth embodiment, the input parameter component canbe either a software input or a hardware input.

In Step 105, Step 205, Step 305 and Step 405, the displayed image can beany of: (1) stereoscopic to achieve re-focusable stereo vision; and (2)conventional non-stereoscopic to achieve re-focusable plain vision. Theimage can be displayed on a display screen that is positioned away fromviewer's body. The image can also be displayed on a wearable displaydevice that is disposed close to viewer's eye or fixed to viewer's head.The image can also be displayed by a scanning light beam projectingdirectly into viewer's eye and forms one or multiple scanning lightspots on the viewer's eye retina, where the fast 2D scan of the lightbeam spot on retina forms perceived image by the viewer. The image canalso be displayed by an MEMS actuated mirror array reflecting one ormore light sources, or an MEMS actuated light source array, whichprojects light beams directly into viewer's eye and forms a 2D imagedirectly on the retina of the viewer's eye.

The recording media of all four embodiments can be any of: (1) an analogor film based media; (2) a digital media, for example a Charge-coupleddevice (CCD) or a Complementary metal-oxide-semiconductor (CMOS) device;and (3) a holographic media.

FIG. 26 is a schematic flow diagram illustrating a feed-back loop thatcan be used during the process to achieve desired focus depth of theviewer for all embodiments.

After initial sensing re-focus intention of viewer at step 261, intendedfocus depth of the viewer is calculated at step 262 with the informationfrom step 261. Then at step 263, either image with corrected focus depthis retrieved from recording media or database as in embodiment 1 andembodiment 2, or new images are generated by adjusting the focus depthadjustment component as in embodiment 3 or by adjusting the focus depthadjustment parameter as in embodiment 4. The retrieved or updated imagefrom step 263 is displayed to the viewer in step 264. Another step ofsensing re-focus intention of viewer happens at step 265. A judgmentstep 266 of whether the desired focus depth has been reached is made byexamining the re-focus information from step 265, wherein if desiredfocus depth is reached, viewer will show no desire to re-focus from step265. Otherwise re-focus intention of viewer will still show in step 265.If desired focus depth is reached, then the re-focus adjustment processends as in step 267. If desired focus depth is not reached, anotherjudgment step 268 is made for whether the re-focus direction from step265 is in the same direction of focusing as in step 261 or not. If there-focus direction is the same, it means prior re-focus adjustment isunder-adjustment and additional re-focus adjustment shall be incrementalfrom the prior adjustment as in 269. Otherwise if the re-focus directionis opposite to step 261 direction, the prior adjustment isover-adjustment and a compensation of the over-adjustment shall be doneas in 2691. Afterwards, the loop repeats from step 262 as describedpreviously.

Such feedback loop can also be used to train the re-focus adjustmentsystem to learn and accommodate each different user's re-focus habit andmake best approach to reach desired focus depth in shortest time andfewest loops.

Before a re-focusable viewing procedure is applied to the viewer'sviewing experience, a training process can be employed to bettercalibrate the viewer's re-focus and vision intention. Images with knownand calibrated different focus depth, or objects with known andcalibrated distance from viewer, can be shown to the viewer. Theviewer's eye-lens information, eye ball position, or brain-wave patternswhen viewing these images at various perceived distances, or objects atvarious spatial distances, from the viewer can be stored as calibrationstandards of the focus depth of this specific viewer. When eye-lens, eyeball position or brain-wave pattern changes during a viewing event ofother images or objects, these previously stored calibration standardscan be used to be compared to such changes and extrapolate the desiredfocus depth.

The training process can be done each time before a re-focusable deviceis initially brought into utilization by a new user. It can also be doneeach time before a re-focusable viewing procedure takes place.

FIG. 27 is a schematic diagram illustrating the application of theinvention in static and motion pictures on a display screen 276. Images277 are displayed on an actual display screen 276 to the viewer 270. Theviewer 270 is mounted with a supporting frame 272 that may contain adata processor 2721 (not shown in FIG. 27) that computes and processesinformation collected by the eye sensor 2711. The processor 2721 canalso be a separate component not on the frame, wherein there is datacommunication between the frame 272 and the processor 2721. Supportingframe 272 supports see-through components 271 which can be composed ofany one or any combination of: (1) sensor 2711 to detect eye-lens focusdepth, or eyeball position at same time; (2) Stereoscopic visionenabling device 2712; and (3) Optional multiplexing component 2713 thatchooses correct focus-depth image from display screen 276. The viewer'sperception of the displayed images 277 through the stereoscopic visiondevice 2712 is 3D objects 275 containing “object 1” and “object 2” atdifferent distances from the viewer. When viewer 270 pays attention to“object 1” and the re-focus intention is sensed by the eye sensor 2711to be upon “object 1”, the 2721 processor processes the eye sensor 2711information and sends a command to the display screen 276 or theoptional multiplex component 2713 to bring “object 1” into focus forviewer 270. “Object 1” is brought into focus in the viewer's vision asrepresented by the solid line, and “object 2” is defocused asrepresented by the dashed line. The viewer's sense of being focusing on“object 1’ can be from changing the displayed images 277 on the displayscreen 276. The viewer's sense of being focusing on “object 1” can alsobe from adjusting a multiplexing component 2713 on the supporting frame272 that only displays the images 277 on the display screen 276 that hascorrect focus depth that focuses on “object 1”, in which case, multiplefocus depth images are shown concurrently on the display screen 276.

An optional camera(s) 273 can be used to record current scene on thedisplay screen 276 to help processor calculate viewer's desired focusdepth. Same re-focus function can also be achieved without thestereoscopic vision, where objects 1 & 2 appear as flat picture insteadof 3D objects in space, but can still be focused upon individually bythe viewer 270. Position, orientation and movement of viewer's head canalso be used as an input parameter when updating the images 277displayed to the viewer 270.

An optical imaging system 18912 with a variable effective focus depthcan be disposed as a part of the components 271 that the viewer's eyessee through, wherein the effective focus depth of the system isautomatically adjusted to the viewer's eye-lens focus depth change inreal-time, such that the images 277 shown focused on same display 276that is at fixed distance from viewer 270 always appear focused on theretina of the viewer's eye at various viewer's eye-lens focus depth. Toviewer 270, the images 277 are at different distances from viewer 270when the focus depth changes in viewer 270's eye. Such optical imagesystem can be any of: a single optical lens with mechanical positioning,a series or an array of optical lenses with mechanical positioning, avariable focus depth optical component that is composed ofelectrically-controlled refractive index material, an optical componentwhose effective optical path for light passing through can be changed byan electrical signal, and an optical component composed based onmicro-electro-mechanical-system (MEMS) actuated lens, mirror or prismarrays.

FIG. 28 is a schematic diagram illustrating the application of theinvention in static and motion pictures with image projector 283. Images285 are displayed to the viewer by an image projector 283 that projectsimage directly into the viewer 280's eye, or by projecting or displayingan image onto a display in front of the viewer 280's eyes where thedisplay is also supported by the supporting frame 282. The viewer 280 ismounted with a supporting frame 282 that may contain computing and dataprocessing components 2821 as shown in FIG. 28. The processor 2821 canalso be a separate component not on the frame 282, wherein there is datacommunication between the frame 282 and the processor 2821.

Supporting frame supports see-through components 271 which can becomposed of any one or any combination of: (1) sensor 2711 that detectseye-lens focus depth, or together with eyeball position; (2)stereoscopic vision enabling device 2712, as well as the image projector283.

The viewer 280 perception of the displayed images 285 through thestereoscopic vision device 2712 is 3D “object 1” and “object 2” atdifferent distances from the viewer 280. When the viewer 280 paysattention to “object 1” and the re-focus intention is sensed by the eyesensor 2711 on “object 1”, the processor 2821 process the eye sensor2711 information and sends command to bring “object 1” into focusedimage for viewer 280. “Object 1” is brought into focus in the viewer280's vision as represented by solid line, and “object 2” is defocusedas represented by the dashed line. The viewer 280's sense of beingfocusing on “object 1” is from changing the displayed images by theimage projector 283.

The image projector 283 can be a wearable display device that isdisposed closed to viewer 280's eye and fixed to viewer 280's head,wherein the display device has an internal image display screen and theviewer 280 sees the display screen through an optical path that makesthe effective optical distance of the image 285 shown on the displayappear at a distance that viewer can comfortably see clearly.

The image projector 283 can also be composed of a device producing ascanning light beam that projects directly into the viewer 280's eyepupil, wherein the scanning light beam projecting directly into viewer'seye forms one or multiple scanning light spots on the viewer 280's eyeretina, where the fast 2D scan of the light beam spot on retina formsperceived image by the viewer.

The image projector 283 can also be composed of a device having an MEMSactuated mirror array reflecting one or more light sources, or an MEMSactuated light source array, which projects light beams directly intoviewer 280's eye and forms a 2D image directly on the retina of theviewer 280's eye.

An optical imaging system 18912 with a variable effective focus depthcan be disposed as a part of the components 271 that the viewer's eyessee through, wherein the effective focus depth of the system isautomatically adjusted to the viewer's eye-lens focus depth change inreal time, such that the image shown focused by image projector 283always appear focused on the retina of the viewer's eye at variousviewer's eye-lens focus depth. Such optical image system can be any of:a single optical lens with mechanical positioning, a series or an arrayof optical lenses with mechanical positioning, a variable focus depthoptical component that is composed of electrically-controlled refractiveindex material, an optical component whose effective optical path forlight passing through can be changed by an electrical signal, and anoptical component composed based on micro-electro-mechanical-system(MEMS) actuated lens, mirror or prism arrays.

Same re-focus function can also be achieved without the stereoscopicvision, where “objects 1” and “object 2” appear as flat picture insteadof 3D objects in space, but can still be focused upon individually bythe viewer 280. Position, orientation and movement of viewer's head canalso be used as input parameter when updating the images displayed tothe viewer 280.

If same part appears in later figures and schematics of this currentinvention without further definition or description, it has the samefunction and definition as described above in FIG. 27 and FIG. 28.

FIG. 29 is a schematic diagram illustrating the application of thecurrent invention for enhanced vision. The application in FIG. 29 issubstantially similar as the application as illustrated in FIG. 27 andFIG. 28 except the following: (1) Images displayed to the viewer aretransmitted from an image or video recording device 294 that recordsfrom a live scene of actual objects 298; (2) Viewer's sense of beingfocusing on “object 1” is achieved by sending a command to the focusdepth adjustment component 296 of the recording device 294 to change theactual focus depth of the recording device 294 so that the “object 1” inactual scene is recorded in-focus as in 299; (3) The image of the livescene of objects 298 is recorded by the recording device 294 withcorrect focus depth reflecting the desired focus depth of the viewer 270or viewer 280, and said recorded image is then sent to be displayed tothe viewer 270 on the display screen 276 as in 292 or sent to bedisplayed to the viewer 280 by the image projector 283 as in 293; (4) Ifthe focus of 296 is not in the desired focus-depth of the viewer, afeedback of desired focus depth is sent back from the frame 272 or frame282 to the recording device 294 as in 295 and 297, to further adjust 296focus depth to reach desired focus depth; (5) The achievable focus depthof the focus depth adjustment component 296 can be different than humaneye, thus an enhanced vision can be realized by enabling viewer 270 orviewer 280 with the ability to have enhanced focus depth capability, andpreferably together with enhanced zoom range at the same time, of liveobjects 298; and (6) Position, orientation and movement of viewer 270 orviewer 280 head can also be used as input parameter when updating theimages displayed to the viewer 270 or viewer 280.

FIG. 30 is a schematic diagram illustrating the application of thecurrent invention for artificial reality. The application in FIG. 30 issubstantially similar as the application illustrated in FIG. 29 exceptthe following: (1) Images displayed to the viewer are generated by animage generation device 3016; (2) Viewer's sense of being focusing onobject 1 is from sending a command to change the focus depth adjustmentparameter 3015 of the image generation device 3016 to change the focusdepth of the generated image 3018, so that the “object 1” in generatedimage 3018 is in focus; (3) Generated image 3018 with desired focusdepth is then sent to be displayed to the viewer 270 on display screen276 as in 3013 or sent to be displayed to the viewer 280 by imageprojector 283 as in 3017; (4) The image scene 3018 and objects do notactually exist in reality, but rather are computer generated artificialobjects, such that the viewer 270 or viewer 280 is viewing an imagescene 3018 that is artificial. With the artificial scene 3018,re-focusable capability and 3D vision, the viewer 270 or viewer 280 canhave an artificial reality experience; (5) If the focus of 3018 is notin the desired focus-depth of the viewer 270 or viewer 280, a feedbackof desired focus depth is sent back from the frame 272 or frame 282 tothe image generation device as in 295 and 297, to further adjust thefocus depth adjustment parameter 3015 to reach desired focus depth ofgenerated image 3018; (6) Interaction between the viewer and theartificial objects can be realized by establishing one or more of otherinput methods into the image generation device 3016, whereas exampleinputs from viewer 270 or viewer 280 are: (a) eye movement; (b) eye lipsmovement; (c) body gesture; (d) body movement; € force exerted by viewer270 or viewer 280 to an external controller device; (f) vocal, opticaland electrical signals initiated by the viewer 270 or viewer 280, toachieve human-machine interaction, whereas camera(s) 3011 and 3012attached to the supporting frame can be used as the gesture and movementcapturing device.

As an example of human-machine interaction: when viewer 270 or viewer280 focuses on “object 1” and “object 1” becomes focused in the view,the viewer can do a gesture to try to rotate “object 1” in space. Thegesture is then captured by camera(s) 3011 and 3012 and sent as an inputsignal into the image generation device 3016. The image generationdevice 3016 then generates new images where the “object 1” being rotatedfrom original orientation to new orientations following viewer 270 orviewer 280 gesture. To the viewer 270 or viewer 280, the “object 1”appears to be rotating in space according to the viewer 270 or viewer280 rotating gesture. During this process, “object 2” position andorientation stays unchanged in the image and to the viewer's perception,since it is not focused upon. Position, orientation and movement ofviewer's head can also be used as input parameter to device 3016 duringthe human-machine interaction.

FIG. 31A and FIG. 31B are schematic diagrams illustrating theapplication of the current invention for augmented reality withartificial objects augmenting viewer interaction with real objects. Theapplication in FIG. 31A and FIG. 31B is substantially similar as theapplication illustrated in FIG. 27 and FIG. 28 except the following: (1)Viewer 270 is viewing real object(s) (not shown in FIG. 31A or FIG. 31B)or objects on real display 276; (2) Imaginary objects 3141 and 3142,which are key pads in FIG. 31A and FIG. 31B, appear to the viewer 270 atspatial positions different than real object(s) or objects on realdisplay 276, where: (a) The imaginary object 3141 can be produced byprojecting a 3D image 314 on a common display where the real objects aredisplayed (FIG. 31A); (b) The imaginary object 3142 can be produced bythe projectors 283 on the supporting frame 282 to the viewer 270 (FIG.31B); (3) The imaginary objects 3141 and 3142 can appear as 3D objectsto viewer 270 by the stereoscopic vision enabling devices on thesee-through components 271; (4) Viewer 270 can re-focus on differentimaginary objects 3141 and 3142 and interact with the objects 3141 and3142 with body gestures, for example “touching” the objects 3141 and3142, and induce a visual or physical response from the real object(s)or objects on real display 276 in view , where: (a) Camera(s) 3111 and3112 on the supporting frame 282 can be used to capture the body gestureof the viewer 270 and measure the position of the body part 3121relative to the intended position of the imaginary objects 3141 and 3142to the viewer 270; (b) With body part position matching position of theimaginary objects 3141 and 3142, and with recognizing viewer's bodygesture, a command can be generated from the gesture and a response canbe made from the real objects or objects on real display 276.

As an example, in the FIG. 31A and FIG. 31B, when the viewer 270 touchesthe imaginary keypads 3141 and 3142 number 3 in viewing space by finger3121 at the spatial position where the number “3” buttons appear to beto the viewer 270, the display 276 will show “1+2=3”. When viewer steersaway eyes from keypad 3141 and 3142, re-focuses and looks at display ata further distance, the keypads 3141 and 3142 can appear as blurred,similar to a real key-pad in same spatial position would appear to theviewer 270, or keypads 3141 and 3142 can also just disappear from theview of viewer 270. Position, orientation and movement of viewer's headcan also be used as input parameter during interaction of viewer 270with the imaginary objects 3141 and 3142.

FIG. 32 is a schematic diagram illustrating the application of thecurrent invention for augmented reality with artificial objectaugmenting real objects. The application in FIG. 32 is substantiallysimilar as the application as illustrated in FIG. 31B except thefollowing: (1) Imaginary objects 3231, 3232 and 3233 are displayed tothe viewer 270 at different spatial positions to the viewer 270, whereimaginary objects can be any of: (a) The imaginary objects 3231, 3232and 3233 are produced by the projectors 283 on the supporting frame 282that project image to the viewer 270; (b) The imaginary objects 3231,3232 and 3233 can appear as 3D objects to viewer 270 by the stereoscopicvision enabling devices on see-through components 271; (c) Imaginaryobjects 3231, 3232 and 3233 as perceived by viewer 270 are at spatialpositions that associated with, and in close proximity to, various realobjects 3221, 3222 and 3223; (2) Viewer 270 can re-focus on differentreal objects 3221, 3222 and 3223, wherein one of the correspondingimaginary objects 3231, 3232 and 3233 associated with each of realobjects 3221, 3222 and 3223 will also appear to be in-focus to viewer270 when the associated real object is in-focus; (3) When a real objectis in focus to the viewer 270, the viewer's focus point is compared tothe physical distance and position of the real objects 3221, 3222 and3223 to the viewer 270. The real object 3221, 3222 or 3223 in focus toviewer 270 will be identified as being at correct position and distancethat matches the viewer 270's intended focus depth and focus point alongthe eye-sight 324 direction. Then an imaginary object 3232 associated tothat real object 3222 being in-focus is also brought into focus inviewer 270's view and at position in proximity to the real object 3222.(4) Viewer 270 can interact with the imaginary objects associated withthe real objects with any or any combination of: (a) body gestures; (b)vocal, electrical, or optical signals, for example viewer 270 “pointingto” the imaginary object 3232 in-focus or speaking out a vocal command,wherein the said signals are acquired by a signal processor in thesupporting frame 282 or a signal processor separated from the supportingframe, and the said signals are interpreted to produce a visual changeof the imaginary object 3232 in view. Camera(s) on the supporting frame282 can be used to capture the body gesture of the viewer 270 andmeasure the position of the body part relative to the imaginary object3232 in-focus or the viewer 270's eye-sight 324 direction. With bodypart position matching the imaginary object 3232 in focus or the viewer270's eye-sight 324 direction, a command can be generated from the bodygesture and a response can be produced by the imaginary objects.Position, orientation and movement of viewer 270's head can also be usedas input parameter when updating the images displayed to the viewer 270.

For example, in FIG. 32, when the viewer 270 focuses on Building 2 of3222, the imaginary box 3232 of “Note 2” appears and in focus to theviewer 270 with physical position appear to viewer 270 to be on top onthe “Building 2” of 3222. “Note 2” 3232 can contain information aboutthe “Building 2” 3222. “Note 1” 3231 on “Building 1” 3221 and “Note 3”3233 on “Building 3” 3223 can appear blurred or entirely invisible tothe viewer 270. When viewer 270 uses a finger to point to the “Note 2”3232 direction, the cameras 3211 and 3212 on supporting frame 282captures viewer 270's hand direction and matches to “ Note 2” 3232direction and makes a change of “Note 2” 3232 appearance as a responseto the gesture.

In this application, the objects 3221, 3222 and 3223 in the actual viewthat the viewer 270 is seeing will form projection image on the retinaof the viewer 270's eye. The object 3222 having the clearest projectionimage or clearer than other objects 3221 and 3223 can also be used toretrieve the information of the focus depth of the lens, and focusingpoint of the viewer 270's sight. For example, “Building 2” 3222 showsclearest image on viewer 270's retina. By identifying this object 3222from the image on the retina and comparing to the image that thecamera(s) 3211 and 3212 capture of the scene that viewer 270 is viewing,the eye-lens focus depth and location of focusing point in the view ofviewer 270's eye can be obtained.

FIG. 33 is a schematic diagram illustrating the application of theinvention for augmented reality with using artificial object to controlreal objects with using viewer's eye or body gestures to interact withthe real object. The application in FIG. 33 is substantially similar asthe application as illustrated in FIG. 32 except the following: (1)Viewer 270 interacts the real object 332, a TV, in FIG. 33, to cause anactual response or action of the real object 332; (2) Viewer 270interaction with the real object 332 is through the imaginary objects333 and 334 that appear and in-focus in viewer's vision when viewer 270focuses on real object 332; (3) Interaction initiated by the viewer 270is in-part by viewer 270 eye gesture, or in some embodiments togetherwith other body gestures of viewer 270. Such eye gestures can be any oneor any combination of: (a) time of stare by viewer 270 on the imaginaryobjects 333 and 334; (b) movement of viewer 270 eyeball; (c) opening andclosing of eye lips of viewer 270 and its frequency; (d) change of eyelips open width; and (e) eye-sight 324 focus point shift in space; (4)Said eye gestures can produce a change of the imaginary objects 333 and334 appearance which leads to a physical response or action of the realobject 332 that is associated with the imaginary objects 333 and 334,wherein such response of the real object 332 can be accomplished bycommunications through signals of any or any combination of: electricalsignal, optical signal, acoustic signal and radio signal, between asignal processor 2721 (not shown in FIG. 33), which processes theviewer's eye information, and in some embodiments, other input signalsfrom viewer 270 body gestures or vocal commands, and the real object332. Said communication between the processor 2721 and the real object332 can also be achieved through a wireless data network or a wirelessdata link.

As an example of this application, when the viewer 270 of FIG. 33focuses on the television 332 and with long enough time staring at thetelevision 332, or by other enabled eye or body gestures, or by vocalsignals, an imaginary menu 333 can appear to viewer 270 in proximity tothe television 332 and lists items that are related to the operation ofthe television 332. When viewer 270's eye-sight 324 focus point shiftsalong the different items of the menu 333, different items can behighlighted in viewer 270's view. Similarly, a sub-menu 334 associatedwith certain menu 333 item, for example “increase/decrease” sub-menu of“Sound” item as in FIG. 33, can appear. With viewer's eye sight 324focus point stays on a given menu item without further shifting for morethan a certain amount of time, or with a subsequent eye gestureinitiated by the viewer, for example a closing and then opening of theeye lips, or by other enabled eye or body gestures, or by vocal signals,a choice of the given menu item where the viewer's eye sight focusesupon is made. Such choice is then processed by the processor 2721 andcommunicated to the television 332 through a data network or a data linkand an action is made to TV 332, for example a decrease of “Sound”volume of TV 332 as in FIG. 33.

FIG. 34 is a schematic diagram illustrating a MEMS actuated micro-mirrorarray used for direct projection of image on the retina of viewer's eye.A collimated beam of projection light 343 with high directionality isprojected upon a mirror array 344. The mirror array 344 can be in theform of mirrors in a one-dimensional array that also scans in thedirection normal to the array formation, or a two-dimensional matrix.Each mirror in the mirror array is actuated by a MEMS based mechanism.The projection light 343 can be produced by a light source of any of,but not limited to: light emitting diode(“LED”), laser diode, solid orgas based laser, fluorescent lamp, and halogen lamp. Each mirror in themirror array 344 is tilted at certain angle to reflect the projectionlight 343 into the pupil of the viewer's eye and through the eye-lens341. With adjusting the angle of tilting of each mirror in the mirrorarray 344, each light beam of the reflected light 345 from each mirrorcan be arranged to pass through the eye-lens 341 at the eye-lens opticalcenter point 3411, which is a preferred scheme of this method. In suchscheme, the reflected light 345 beams effectively concentrate on theoptical center point 3411. With reflected light 345 passing through theoptical center point 3411, the refraction by eye-lens of the reflectedlight 345 is minimal and reflected light 345 enters the eye in the formof a straight light beam with minimal distortion. When reflected light345 from each mirror reaches the retina 342 of the viewer's eye, a lightspot is created, which is then regarded as a pixel 346 projected by thecorresponding mirror of mirror array 344 of the projection light 343.During operation, each mirror of the mirror array 344 produces adifferent pixel 346 on the retina. With all pixels combined, an imagecan be effectively created on the retina by the mirror array.

Compared to prior arts, which uses single 2-D scanning mirror to projectlaser beam onto retina to produce image, this new method as shown inFIG. 34 with using mirror array relieves the concern of permanent retinadamage in the case of a malfunction. In prior arts using single mirrorscanning method, since a single light beam power is effectively spreadinto a larger area on the retina during scan to produce image, arealight power density on the retina can be small enough to not cause anydamage to the retina. During malfunction, if the mirror stops moving andall light power is then focused on a single spot on the retina, retinadamage is then very likely. In fact, this possibility of retina or eyedamage is one limiting factor of the prior arts adoption into commercialuse

For the method as in FIG. 34, incoming projection light 343 intensity isalready spread through all mirrors of the mirror array 344, with eachmirror only producing a light spot or pixel 346 on the retina 342 with asmall portion of the total light power of the projection light 343.During malfunction, even if the mirrors stop moving, the light pixels346 on the retina 342 stays spread out and thus damage by the focusedlight energy as in prior art with single 2-D scanning mirror can beavoided. Mirrors of 344 can reflect incoming light 343 to project uponretina 346 in sequential order, thus only one or a few mirrors reflectlight beam passing through eye-lens 341 and projecting upon 346 at anygiven instant time, thus further reducing risk of eye damage.

A second advantage of the method of FIG. 34 is the speed of imagerefreshing is much faster than in prior art of single 2-D scanningmirror. In prior art of single 2-D scanning mirror, an image isrefreshed at the max speed of the single scanning light beam finishesscanning of the whole image. While in the new method of FIG. 34, theimage is refreshed at the max speed of changing the angle of a singlemirror, whereas all mirrors of the mirror array 344 can be updated oftheir angular positions in a single step, which is much faster thanscanning a single 2-D mirror to produce an entire image.

A third advantage of the new method is the ability to achievewider-viewing angle and higher resolution than prior art. The mirrorarray 344 can be formed on a curved substrate such that high anglereflection of the incoming projection light 343 by edge mirrors ofmirror array 344 can be achieved, and produce wide-viewing-angle imageon the retina 342. For prior art, largest viewing angle is limited bythe MEMS mechanism and the maximum tilting angle of the mirror. Sincethe light beam of the projection light 343 is only required to be a wideand collimated light, and the mirror size of the mirror array 344determines the reflected beam size and eventually the pixel 346 size onretina 342, with advanced lithography and manufacturing techniques, themirror size and pixel 346 size can reach micron-level or smaller,approaching or exceeding the detection resolution of the retina of ahuman eye. For prior art single mirror scanning method, due to safetyconcern as well as scanning speed and laser system limitations,micron-size light beam is not applicable to achieve the function ofdirect projection imaging.

The method as in FIG. 34 can have any one or any combination of belowfeatures: (1) The driving mechanism of the mirrors in the mirror array344 can be any of: MEMS, magnetic force, piezoelectric effect,electrostatic force, capacitive force, or thermally induced shapechange; (2) The reflected light 345 beams effective concentration pointcan be any of: eye-lens optical center 3411, between eye-lens opticalcenter 3411 and cornea, in front of cornea and outside the eye, insidethe eye and at position between the eye-lens optical center 3411 andretina 346, wherein the concentration point can be either a focus pointof light beam 345 or a point of smallest light beam 345 size; (3) Theprojection light 343 can be alternating between various wavelength,wherein at each different wavelength, the mirrors of the mirror array344 change to a different set of angle positions, such that imageprojected on retina is perceived as a color image by viewer; (4) Therecan be multiple projection light 343 sources projecting on the mirrorarray 344, with each projection light 343 source having a differentlight wavelength, or different color. The mirror array 344 can havemultiple subsets of mirrors with each subset of mirrors reflecting eachof the multiple light 343 sources and produces multiple images ofdifferent colors overlapping on the retina to form a colored image. Themirrors of mirror array 344 can be readily replaced with optical prismsto achieve sample reflection function without limitation.

Mirrors of mirror array 344 can be projecting pixels 346 on the retina342 with different timing instead of projecting all pixelssimultaneously, so that high local light intensity of the effectivefocus point of the reflected light 345 can be reduced to avoid damage toeye tissue.

FIG. 35 is a schematic diagram illustrating the micro-mirror array ofFIG. 34 being implemented with input from viewer's eye information toaccommodate the viewer's eye-lens change and project image in focus onretina at varying eye-lens focus depth. Similar as described in FIG. 34,a two-dimensional mirror array 354 reflects projection light 353 by eachmirror of the mirror array 354. Reflected light 355 passes through theeye-lens and produces a projected image 356 on the retina 352.

All specifications and descriptions of the mirror array 354, eye-lens351, retina 352, reflected light 355, projection light 353, andprojected image 356 are similar as the mirror array 344, eye-lens 341,retina 342, reflected light 345, projection light 343, and projectedimage 346 in FIG. 34.

However, FIG. 35 shows additional components including optical emitter357 and optical detector 358, which are used to detect the eye-lenschange and pupil position change as described in FIG. 12A through FIG.12H. The optical signal containing eye-information change regardingeye-lens and pupil is sent to a computing device 359 as shown by 3591.The computing device 359 calculates the desired focus depth from thesensed eye-information and produced updated version of the image 356 tobe projected on retina 352, which reflects the desired focus depth ofthe viewer. The updated image is sent from computer device 359 to themirror array 354 controller as shown by 3592. The mirror array 354 thenchanges accordingly to project updated image with any change of: imageshape, size, form, color, contrast, brightness or other opticalproperties to produce an effective change of viewer's perception thatfollows the focus depth change of the eye-lens of the viewer's eye.

For example, when viewer tries to see clearly of an originallyde-focused first object of many objects in the projected image 356 andchanges the eye-lens to try to focus on the first object, the mirrorarray changes accordingly such that the first object appears clearlyfocused in the projected image 356 with a final form that reflectsintended perceived spatial position of said first object having afocused image, to give the viewer a sense of 3D space and ability offocusing on the objects into the 3D space.

FIG. 36A and FIG. 36B then show alternative possible image projectionlight paths than the light path produced by the mirror array 354 asshown in FIG. 35. All specifications and descriptions of the mirrorarray 364, eye-lens 361, retina 362, reflected light 365, and projectedimage 366 are similar as the mirror array 354, eye-lens 351, retina 352,reflected light 355, and projected image 356 in FIG. 35. In FIG. 36A,the reflected light 365 converges towards an area or a point 368 behindthe eye-lens 361 within the eye before entering the eye-lens 361. Thenreflected light 365 of FIG. 36A is diffracted by the eye-lens 361 andproduces image 366 upon the retina 362. In FIG. 36B, another light pathis shown where the reflected light 365 is converged towards an area or apoint 369 in front of eye-lens 361, where 369 is preferred to be inclose proximity to the opening the pupil of the eye, before entering theeye-lens 361. Then reflected light 365 of FIG. 36B is diffracted by theeye-lens 361 and produces image 366 upon the retina 362. The differentlight paths of FIG. 36A and FIG. 36B from FIG. 35 are achievable byadjusting mirror array 364 mirror angle arrangement and distance ofmirror array 364 from eye-lens 361.

The mirror arrays 344, 354 and 364 as shown in FIG. 34, FIG. 35, FIG.36A and FIG. 36B reflect incoming light 343 and 353, which is preferredto have high directionality, to produce projected images on the retinaof the eye. Alternatively, an array of light emitting devices that canproduce and project high directionality light beams directly fromdifferent angles into the eye can also achieve the same direct retinaimage projection without the need of an incoming light beams 343 or 353as in FIG. 34 and FIG. 35.

FIG. 37A shows a first example of a substrate 3701 supporting lightemitting devices 3702 in the form an array, for the purpose of replacingthe mirror array 344, 354, 364 as in FIG. 34—FIG. 36B by directlyproducing an array of high directionality light beams to project imageon the retina of the viewer's eye. The substrate 3701 with the lightemitting devices 3702 shall replace the incoming light 343, 353 andmirror arrays 344, 354 as in FIG. 34 and FIG. 35. In FIG. 37A, an arrayof light emitting devices 3702 are disposed upon the substrate 3701.Each of the device 3702 is producing a light beam 3703. For the purposeof direct image projection on the retina of the eye, the light beam 3703is preferred to be sufficiently directional, such that each of the lightbeam 3703 from a given light emitting device 3702 produces a specificlight spot upon the retina of the eye which constitute a pixel of theimage that is to be projected upon the retina. The light emittingdevices 3702 can by any of, but not limited to, light emitting diodes(LED), organic light emitting diode (OLED), and laser diode. The lightemitting devices 3702 may have certain built-in optical cavity structurethat is manufactured together with the light emitting device 3702,especially when 3702 is an LED or OLED device, to help promote selectedoptical modes of the light beam 3703 emitted from the device 3702, suchthat the emitted light beam 3703 is highly directional.

For typical LED or OLED device 3702, due to the light beam 3703 beingmostly diverging when directly coming out of the LED or OLED device3702, light beam shaping components can be placed on the LED or OLEDdevices 3702. FIG. 37B shows an example of collimation lens 3704 thatcan be used to collimate and converge the light beam 3703 to achievehigh directionality of the light beam 3703. FIG. 37C then shows anexample of optical aperture 3705, for example an optical crystal, asection of optical waveguide or optical fiber, being disposed inimmediate proximity to the LED or OLED 3702. The optical aperture hasthe function of selecting light beam 3703 components that is highlydirectional and allows the passage of such components towards the eye.The collimation lens function of 3704 in FIG. 37B and optical aperturefunction of 3705 may also be integrated into a single light beam shapingcomponent. Lens 3704 and optical aperture 3705 are preferred to bemanufactured and integrated with each LED or OLED device 3702 of thearray on the substrate 3701 during a single manufacturing process.

The emitted light beam 3703 angle adjustment can be achieved by asurrounding or underlying mechanical angle adjustment mechanism attachedto each individual light emitting device 3702. The angle adjustmentmechanism is not shown in FIG. 37A-FIG. 37C, and such mechanism can bebased on any of: MEMS, magnetic force, piezoelectric effect,electrostatic effect, capacitive effect, or thermally induced shapechange.

The light beam 3703 angle adjustment from the light emitting devices3702 can also be achieved by a global curvature change of a flexiblesubstrate 3701 wherein the light emitting devices 3702 are disposed uponor embedded within the substrate 3701. In one embodiment, the substrate3701 is a flexible substrate whose surface curvature can be changed uponan externally exerted force. In another embodiment, the substrate 3701curvature can be change by electrically controlled mechanisms connecteddirectly to, or embedded within, the substrate 3701, wherein suchelectrically controlled mechanism can be any of: MEMS, magnetic force,piezoelectric effect, electrostatic force, capacitive force, orthermally induced shape change. In one embodiment, the light emittingdevices 3702 are OLED devices that can be manufactured on a flexiblesubstrate. Light beam 3703 from each OLED 3702 device is preferred to bein the direction that is normal to the surface of the substrate 3701,such that when curvature of the substrate 3701 changes, light beam 3703directions at various locations across the substrate 3701 maintain theirnormal directions to the substrate 3701 surface. The normal direction ofthe light beam 3703 to the flexible substrate surface 3701 can befurther assisted or maintained by the light beam shaping components 3704or 3705 as in FIG. 37B and FIG. 37C, which are disposed immediatelyadjacent to the OLED device 3702 on the flexible substrate 3701 andconform to the curvature of the substrate 3701.

Another alternative method to achieve the same direct retina imageprojection without the need of the mirror arrays 344, 354 and 364 as usein FIG. 34, FIG. 35, FIG. 36A and FIG. 36B is to use an array of lightpassage components, for example, an array of micro optical waveguides,or an array of optical fiber sections, or an array of micro opticallens, to create high directionality light beams to project image on theretina. In this new method, the incoming light beams 343 and 353 as inFIG. 34 and FIG. 35 will now be directed to shine through the array ofthe light passage devices towards the eye pupil, wherein the array oflight passage devices allow the incoming light to pass or transmitthrough at the locations of the light passage devices. FIG. 38 shows anexample of the alternative new method, where all specifications anddescriptions eye-lens 381, retina 382, and projected image 386 aresimilar as eye-lens 351, 361, retina 352, 362, and projected image 356,366 as in FIG. 35-FIG. 36B. Different than in FIG. 35-FIG. 36B, themirror arrays 354, 364 of FIG. 35-FIG. 36B are replaced by an array oflight passage components 384. The incoming light 383 is directed towardsthe eye-lens and the light beams that pass through the light passagecomponent array 384, i.e. transmitted light 385, are directed in variousdirections such that an image 386 can be produced on the retina 382.

FIG. 39A shows a first example of a substrate 3901 supporting lightpassage components 3902 in the form an array, which constitutes thearray 384 as in FIG. 38. Preferably, a light emitting device 3900projecting light towards the substrate 3901 is in close proximity to thesubstrate 3901. Each of the light passage components 3902 has a lightintake end that is exposed on one surface of the substrate 3901 facingthe light emitted device 3900 to allow the light emitted from the device3900 to enter the light passage components 3902. Each of the lightpassage components 3902 has a light output end exposed on the opposingsurface of the substrate 3901 facing the pupil of the eye, where thelight beams 3903 exiting the light passage components 3902 have highdirectionality to produce projected image on the retina of the eye.Substrate 3901 with the light passage components 3902, and the lightemitting device 3900 shall replace the incoming light 343, 353 andmirror arrays 344, 354 as in FIG. 34 and FIG. 35.

For the purpose of direct image projection on the retina of the eye, thelight beam 3903 is preferred to be sufficiently directional, such thateach of the light beam 3903 coming out from a given light passagecomponent 3902 produces a specific light spot upon the retina of the eyewhich constitutes a pixel of the image that is projected upon theretina. The light passage components 3902 can be any of, but not limitedto, micro optical waveguides, optical fiber sections, or micro opticallens, to create high directionality light beams. The light emittingdevice 3900 can be composed any of, but not limited to, LED, OLED, laserdiode, fluorescent lamp. The light produced by the light emitting device3900 can be a directional light shining towards the substrate 3901, oran ambient light with low directionality which when passes through thelight passage components 3902 obtains high directionality. The lightpassage components 3902 may have certain partial reflective coating atits light intake and output ends, such that the partial reflectionbetween the two ends within the body of the light passage components3902 forms an optical cavity that can help promote selected opticalmodes of the light beam 3903 to be emitted from the component 3902towards the eye. One such promoted optical mode may make the emittedlight beam 3903 being highly directional.

For the case where the light beam 3903 being still diverging when comingout of the output end of the light passage components 3902, light beamshaping components can be placed immediately next to the light passagecomponents 3902. FIG. 39B shows an example of collimation lens 3904 thatcan be used to collimate and converge the light beam 3903 to achievehigh directionality of the light beam 3903. FIG. 39C then shows anexample of optical aperture 3905, for example an optical crystal,optical waveguide or optical fiber, being disposed in immediateproximity to the light passage components 3902. The optical aperture3905 has the function of selecting light beam 3903 components that ishighly directional and allows the passage of such components towards theeye. The collimation lens function of 3904 in FIG. 39B and opticalaperture function of 3905 may also be integrated into a single lightbeam shaping component. Lens 3904 and optical aperture 3905 arepreferred to be manufactured and integrated with each light passagecomponent 3902 on the substrate 3901 during a single manufacturingprocess.

The emitted light beam 3903 angle adjustment can be achieved by asurrounding or underlying mechanical angle adjustment mechanism attachedto each individual light passage component 3902. The angle adjustmentmechanism is not shown in FIG. 39A-FIG. 39C, and such mechanism can bebased on any of: MEMS, magnetic force, piezoelectric effect,electrostatic force, capacitive force, or thermally induced shapechange.

The light beam 3903 angle adjustment from the light passage components3902 can also be achieved by a global curvature change of a flexiblesubstrate 3901 wherein the light passage components 3902 are embedded inthe substrate 3901. In one embodiment, the substrate 3901 is a flexiblesubstrate whose surface curvature can be changed upon an externallyexerted force. In another embodiment, the substrate 3901 curvature canbe change by electrically controlled mechanisms connected directly to,or embedded within, the substrate 3901, wherein such electricallycontrolled mechanism can be any of: MEMS, magnetic force, piezoelectriceffect, electrostatic force, capacitive force, or thermally inducedshape change. In one embodiment, the light passage components 3902 aremanufactured and embedded into a flexible substrate 3901. Light beam3903 exiting from each light passage component 3902 is preferred to bein the direction that is normal to the surface of the substrate 3901,such that when curvature of the substrate 3901 changes, light beam 3903directions at various locations across the substrate 3901 maintainstheir normal directions to the substrate 3901 surface. The normaldirection of the light beam 3903 to the flexible substrate surface 3901can be further assisted or maintained by the light beam shapingcomponents 3904 or 3905 as in FIG. 39B and FIG. 39C, which are disposedimmediately adjacent to the light passage components 3902 on theflexible substrate 3901 and conform to the curvature of the substrate3901.

The array of light emitting devices of FIG. 37A-FIG. 378C, and the arrayof light passage components of FIG. 39A-FIG. 39C, can both replace themirror arrays of the FIG. 35-FIG. 36B and produce light beams followingsimilar light paths of converging behind, within or in front of the eyelens to project image upon the retina as shown in FIG. 35-FIG. 36B.

FIG. 40A and FIG. 40B show an example of a flexible substrate withembedded OLED devices being used to project images on the retina of theeye at different focus depth of the eye-lens with changing the curvatureof the flexible substrate, during the event of the eye-lens focus depthchange by the viewer intention. However, specifications of this exampledo not limit the application of the general concept of utilizingflexible substrates with embedded light emitting devices in ways thatare different than as exactly described in FIG. 40A or FIG. 40B. Duringdescription of FIG. 40A and FIG. 40B, the eye-lens 401 will be treatedas a typical optical lens.

In FIG. 40A, OLED devices 404 are arranged in an array, preferably inthe form of a two-dimensional matrix, across the substrate 403 surfacefacing the eye-lens 401 of the viewer's eye. OLED devices 404 arearranged in such a way that light beams 405 emitted from the OLEDdevices 404 are normal to the surface curvature of the substrate 403 ateach OLED device 404 location. Light beam shaping components, forexample 3704 and 3705, as shown in FIG. 37B and FIG. 7C may also existon the substrate 403 in immediate proximity to each of the OLED device404. In the case of the FIG. 40A, light beams 405 that are projectedfrom the substrate 403 surface towards the eye-lens 401 converge towardsthe focal point 407 of eye-lens 401 at the back of the eye-lens 401,which is between the eye-lens 401 and retina 402. According to principleof optics, after passing through the eye-lens 401, the light beams 405are diffracted into parallel light beams 406 that produce an image onthe retina 402 of the viewer's eye, with each light beam 406 projectedupon the retina as a pixel of the projected image.

Designating the two outmost light beams of 406 of FIG. 40A projectingthe light spots on the retina 402 being the boundary of the projectedimage, when eye-lens 401 changes shape, for example as in FIG. 40B whenthe viewer intends to focus on an object that is closer to the viewer,i.e. focus depth of viewer changes and becomes shorter than in FIG. 40A,the eye-lens 401 is compressed in the vertical length direction andincreases in the horizontal width, such that the focal point 407 movescloser to the eye-lens 401 center as in FIG. 40B. To maintain theprojected image size and field of view on the retina 402, the lightbeams 405 are needed to have a larger incoming angle 4052 between thetwo outmost light beams of 405 as shown in FIG. 40B. Light beams 405still converge towards the focal point 407 at its closer to eye-lens 401location. Since the light beams 405 projected by OLED devices 4042 arenormal to the substrate surface 403, the light beams 405 are nowrequired to be projected by another set of OLED devices 4042 that aredifferent from the set of OLED devices 404, where outmost light beams of405 are now emitted from OLED devices 4042 that are closer to thesubstrate 403 edges. At the same time, the substrate 403 needs toincrease its curvature and its center distance from the eye-lens 401relative to its original curvature and position as labeled 4032 in FIG.40B, to a new position as shown by substrate 403 to achieve the requiredprojected image on the retina 402 at the changed focus depth of theeye-lens 401.

Besides the substrate curvature and position change to achieve a globallight beam angle re-orientation of the incoming light beams 405 as shownin FIG. 40A and FIG. 40B, to project image on the retina 402 atdifferent focus depth of the eye-lens 401, one or more of the OLEDdevices 404 or 4042 may also have a surrounding or underlying mechanicalangle adjustment mechanism that is (1) attached to each of these one ormore OLED devices 404 or 4042, or (2) attached to any light beam shapingcomponents disposed in close proximity to each of the OLED device totransmit the light beams as described in FIG. 37B-FIG. 37C, or (3)attached to any reflective optical components, for example mirrors,disposed in close proximity to each of the OLED device to reflect thelight beams, and thus adjust the light beams emitted from these OLEDdevices 404 or 4042, for fine angle adjustment. The angle adjustmentmechanism can help accomplish faster and more accurate light beam anglecontrol, or in some embodiments provide light beams at angles that arenot achievable by global angle adjustment with the flexible substratecurvature change alone. In some embodiment, for a 2-dimensional matrixof the OLED array, the flexible substrate can be used to adjust lightbeams angles in a first angular freedom direction and the angleadjustment mechanism is used to adjust light beams angles in the secondangular freedom direction orthogonal to the first direction, such that acombined function of flexible substrate and angle adjustment mechanismachieves an effective focus of the light beams at various locations inspace. The angle adjustment mechanism is not shown in FIG. 40A and FIG.40B, and such mechanism can be based on any of: MEMS, magnetic force,piezoelectric effect, electrostatic force, capacitive force, orthermally induced shape change. Even though the FIG. 40A and FIG. 40Bare described with assuming all light beams 405 being emitted from OLEDdevices 404 or 4042 are normal to the substrate surface, in someembodiments, one or more of the OLED devices 404 or 4042 may emit lightbeams 405 in directions not normal to the surface of the substrate. Forexample, the OLED devices 404 or 4042 that are closer to the edge of thesubstrate 403 may emit light beams at angles not normal to thesubstrate, to achieve a better quality projected image on the retina 402or to achieve a wider field of view that the viewer perceives. The OLEDdevices 404 or 4042 emitting light beams 405 not being normal to thesubstrate surface can be an inherent feature that is built into the OLEDarray 404 or 4042 on the substrate 403 during manufacturing, or achievedby the angle adjustment mechanisms as aforementioned.

Flexible substrate method as described in FIG. 40A and FIG. 40B is basedon OLED devices array. For the same purpose of projecting image atdifferent eye-lens focus depth, the same concept of adjusting thecurvature and position of flexible substrate as described in FIG. 40Aand FIG. 40B can be readily combined with any array of light emittingdevices or array light passage components as described in FIG. 37A-FIG.39C without limitation and with necessary adjustments as described inFIG. 37A-FIG. 39C.

FIG. 41A, FIG. 41B and FIG. 41C describe the embodiment of applicationof the flexible substrate 416 to project image on retina 413 of theeyes, left eye 411 and right eye 412, of the viewer. FIG. 41A shows thecase when the viewer is looking at and focusing on a far object. In FIG.41A, the eye sights of the two eyes 411 and 412 are sufficientlyparallel. One flexible substrate 416 in the curved form and containinglight emitting devices or light passage components is positioned inproximity to the pupil 414 of each eye. A housing 417 providing fixtureand support for each of the substrate 416 exists on the opposing side ofeach of the substrate 416 relative to the eyes 411 and 412. The housing417 maintains its position and is sufficiently stationary relative toeach of the eye 411 and 412. The connections 418 as shown in FIG. 41Abetween each pair of substrate 416 and housing 417 represent thesubstrate curvature adjustment mechanism that is capable of changing thecurvature of the substrate 416 and the position of the substrate 416relative to the pupil 414 of each eye 411 and 412 during the event ofeye sight direction change or eye-lens focus depth change. In the casewhen light passage components array is embedded in the substrates 416 toproduce directional light beams, light emitting devices 3900 asdescribed in FIG. 39A-FIG. 39C can be disposed in between the substrate416 and housing 417 for each eye. Light beams 4151 from the substrate416 facing the left eye 411, produced by light emitting devices or lightpassage components embedded substrate 416, shine into left eye 411through pupil 414 to project image on the retina 413 of left eye 411,while light beams 4152 from the substrate 416 facing the right eye 412,produced by light emitting devices or light passage components embeddedon substrate 416, shine into right eye 412 pupil 414 to project image onthe retina 413 of right eye 412.

In FIG. 41B, both eyes 411 and 412 rotate to the right side when theviewer is looking at and focusing on a far object to the right side ofthe viewer. In FIG. 41B, the eye sights of the two eyes 411 and 412 arestill sufficiently parallel due to the object being far, but both eyesrotate to the right side of the substrate 416. With detection of sucheye-sight angle change and any focus depth change, the substrate 416curvature are adjusted by the adjustment mechanism 418. FIG. 41B showsthe right side connections 418 connecting to each substrate 416 beingelongated and the left side connections 418 connecting to each substrate416 being shrunk in length, and the substrate 416 facing each eye showsan increased curvature on the right side and a reduced curvature of theleft side of each of the substrate 416. Meanwhile, light beams 4151 and4152 shining into each eye 411 and 412 are produced by a new set oflight emitting devices or light passage components closer to the rightside of the substrate 416 for each eye. With the aforementionedsubstrate 416 curvature change and re-arrangement of light beam emittinglocations from the substrate 416 as in FIG. 40B, image projection onretina 413 of each eye can be achieved with desired perception by theviewer in the case of FIG. 41B when the viewer rotate the eyes to focuson a right side far object.

FIG. 41C then shows the case when the viewer is looking at and focusingon a near object, where the eye sights of the two eyes 411 and 412 areconverging to a point at the center, due to the object being close andcentered. With detection of such eye-sight angle change and focus depthchange, the substrate 416 curvature are adjusted by the adjustmentmechanism 418 for both eyes following the process similar to what isdescribed in FIG. 40B when focus depth becomes shorter. FIG. 41Cdescribes such curvature change of substrate 416 by showing the left eye411 rotating to the right, while the substrate 416 facing the left eye411 shows the section of the substrate 416 producing the light beams4151 having an substantially increased local curvature. At the sametime, the connections 418 are shortened at the section of the substrate416 producing the light beams 4151. With a larger substrate 416curvature and increased distance from left eye pupil 414 at the sectionof the substrate 416 producing the light beams 4151, same as describedin FIG. 40B, light beams 4151 project image on the retina 413 of theleft eye 411 to achieve a correct perception by the viewer to look at aclose object. Similarly, for right eye 412 that rotates to the left,substrate 416 facing the right eye 412 and connections 418 undergo sameadjustment such that the light beams 4152 from the section of thesubstrate 416 can project image on the retina 413 of the right eye 412to achieve a correct perception by the viewer.

The substrate adjustment mechanism 418 as shown in FIG. 41A-FIG. 41C canbe based on any one or any combination of: MEMS, magnetic force,piezoelectric effect, or thermally induced shape change, memory alloy,artificial muscle, and air pressure.

FIG. 34-FIG. 41C show various embodiments of methods to utilize array ofmirrors, array of light emitting devices, and array of light passagecomponents to produce image directly on the retina of the eye. However,in practice by those skilled in the art, between the various arrays andthe eye, other optical components, which can include any of: opticallens, mirrors, prisms, optical filters, optical polarizers, and opticalshutters, can be used to control the light beams going into the eye tohave the proper configuration to produce desired image. However, thefundamental function of using individual light beam angle adjustment byvarious described mechanisms, or using global light beam angleadjustment by a flexible substrate, or a combination of the two, is themajor function that enables the viewer to see objects far and near witha desired perception from the images projected on the retina whenviewer's viewing angle and focus depth change.

FIG. 42A through FIG. 42E then show possible implementations of variousmechanical angle adjustment mechanisms as aforementioned from FIG. 34through FIG. 41C. These examples do not limit the implementation ofthese mechanisms in ways that are different than as exactly described,but following same demonstrated principles.

FIG. 42A shows a light output device 4202, which can be a mirrorreflecting an incoming light (not shown in FIG. 42A), a light emittingdevice, or a light passage device passes an incoming light (not shown inFIG. 42A). Light output device 4202 outputs a directional light beam4203. Light output device 4203 is disposed on a rotational platform 4201that has a rotational hinge point 4204. When platform 4201 rotatesaround the hinge point 4204 as indicated by arrow 4205 by a pushing orpulling force exerted on the fixture 4206 that is attached to theplatform 4201, the output device 4202 rotates with platform 4201 and thedirectional light beam 4203 also changes its direction. FIG. 42A showsthe case when the force exerted on fixture 4206 is by a MEMS device,which contains at least one pair of MEMS actuation arm sets, where armset 42072 is stationary and arm set 42071 is attached to fixture 4206.When the arm set 42071 and arm set 42072 are with various voltage orelectrostatic charges, due to electrostatic effect between the arms of42071 and arms of 42072, the arm set 42071 can move relative to thestationary arm set 42072 and exerts push or pull force on the fixture4206. Subsequently, the light beam 4203 angle is changed.

FIG. 42B is identical to FIG. 42A except that the fixture 4206 isattached to a permanent magnet 42082, with 42083 arrow designating themagnetization direction of the magnet 42082. 42081 is an electric coildisposed in proximity to the magnet 42082. When current is appliedthrough the coil 42081 with various current flow directions and currentstrength, the coil 42081 generates different magnetic field whichsubsequently pulls or pushes the magnet 42082 and the attached fixture4206 to change the light beam 4203 angle.

FIG. 42C is identical to FIG. 42A except that the fixture 4206 isattached to an electrode 42092 of a capacitive device, which containsanother stationary electrode 42091. When opposing electric charges arestored on electrodes 42091 and 42092, the two electrodes attract eachother. When same electric charges are stored on plates 42091 and 42092,the two electrodes repel each other. Alternatively, if the capacitivedevice is used as a capacitor, where electrodes 42091 and 42092 storeopposite charges, and the capacitor is being charged by an electricalsource, due to capacitive effect the electrodes 42091 and 42092 willexpel each other during the charging process. If the capacitor is beingdrained of stored charges, due to the capacitive effect the twoelectrodes 42091 and 42092 will attract each other. During the expel andattraction process of the electrodes, electrode 42092 can move relativeto the stationary electrode 42091, which subsequently pulls or pushesthe fixture 4206 to change the light beam 4203 angle.

FIG. 42D is identical to FIG. 42A except that the fixture 4206 isattached to a surface 42101 of a piezo element 42100. When a voltage isapplied between the surface 42101 and the surface 42102 that is on thesurface of the piezo element 42100 opposing the surface 42101, the shapeof the piezo element extends or contracts in its shape between thesurfaces 42101 and 42102 depending on the direction and amplitude of thevoltage. Alternatively, for some other type of piezo element, a voltagecan be applied between opposing surfaces of the piezo element 42100 thatare normal to the surface of 42101 to induce a piezo element shapechange between surfaces 42101 and 42102. When the surface 42102 is fixedon a stationary substrate, or the piezo element 42100 is fixed in space,the piezo element 42100 shape change subsequently pulls or pushes theattached fixture 4206 to change the light beam 4203 angle.

FIG. 42E is identical to FIG. 42A except that the fixture 4206 isattached to a thermally induced shape change element 42201, where thefree end of the element 42201 is attached to fixture 4206, and a distalend of the element 42201 is fixed on a stationary substrate. Whenelement 42201 is heated or cooled, the shape of element 42201 willchange by extending or contracting between the free and the distal ends.Such heating or cooling can be through applying various current flowingthrough the element 42201, where the current may heat or cool theelement depending on the material, composition and structure of element42201. When the element 42201 shape changes, it will subsequently pullsor pushes the attached fixture 4206 to change the light beam 4203 angle.

While the current invention has been shown and described with referenceto certain embodiments, it is to be understood that those skilled in theart will no doubt devise certain alterations and modifications theretowhich nevertheless include the true spirit and scope of the currentinvention. Thus the scope of the invention should be determined by theappended claims and their legal equivalents, rather than by examplesgiven.

What is claimed is:
 1. A method to achieve virtual reality comprising:providing a display device and an eye sensor; displaying a virtual sceneto a viewer by said display device; using said eye sensor to detect anintention of said viewer to focus on a first object in said virtualscene; and updating said virtual scene by rendering said first objectappearing focused, and at least one second object appearing defocused,to said viewer, thereby satisfying said intention.
 2. The methodaccording to claim 1, wherein said intention is determined by acomputing device utilizing data obtained from said eye sensor and dataof said virtual scene.
 3. The method according to claim 1, wherein saideye sensor comprises an optical emitter and an optical detector.
 4. Themethod according to claim 1, wherein said display device is one of: ascreen of an electronic device; a screen for image projection; ahead-mounted display; an micro-electro-mechanical-system (MEMS) mirrorarray reflecting at least one incident light to project image ontoretina of said first eye; a flexible substrate having a variablecurvature and containing an array of light emitting devices; a flexiblesubstrate having a variable curvature and containing an array of lightpassage components (LPCs), wherein said LPCs are any of: are any of:micro optical waveguides, optical fiber sections, or micro optical lens.5. The method according to claim 1, wherein said intention is detectedby said eye sensor from an eye of said viewer with detecting any of: eyelens focus depth; pupil position.
 6. The method according to claim 3,wherein said optical emitter emits light towards an eye of said viewer,whereas said optical detector detects property of said light reflectedfrom said eye including: dispersion of said reflected light, duration ofpulse of said reflected light, delay between pulses of said reflectedlight, and direction of said reflected light.
 7. The method according toclaim 1, wherein said eye sensor is embedded in a see-through substratethrough which said viewer is displayed said virtual scene.
 8. The methodaccording to claim 7, wherein said see-through substrate is in contactwith cornea of an eye of said viewer.
 9. A method to achieve augmentedreality comprising: providing an image capture device, a display deviceand an eye sensor; capturing a real scene of a viewing space of a viewerby said image capture device; displaying a virtual scene to said viewerby said display device, whereas said virtual scene overlaps said realscene in said viewing space of said viewer; wherein said real scenecontains a first real object and a second real object; wherein saidvirtual scene contains a first virtual object relating to said firstreal object and a second virtual object relating to said second realobject; using said eye sensor to detect an action of said viewer tofocus on said first real object in said real scene; and updating saidvirtual scene by rendering said first virtual object appearing focused,and said second virtual object appearing defocused, to said viewer. 10.The method according to claim 9, wherein said action is determined by acomputing device utilizing data obtained from said eye sensor and dataof said real scene obtained from said image capture device.
 11. Themethod according to claim 9, wherein said eye sensor comprises anoptical emitter and an optical detector.
 12. The method according toclaim 9, wherein said display device is one of: a screen of anelectronic device; a screen for image projection; a head-mounteddisplay; an micro-electro-mechanical-system (MEMS) mirror arrayreflecting at least one incident light to project image onto retina ofsaid first eye; a flexible substrate having a variable curvature andcontaining an array of light emitting devices; a flexible substratehaving a variable curvature and containing an array of light passagecomponents (LPCs), wherein said LPCs are any of: are any of: microoptical waveguides, optical fiber sections, or micro optical lens. 13.The method according to claim 9, wherein said action is detected by saideye sensor from an eye of said viewer with detecting any of: eye lensfocus depth; pupil position.
 14. The method according to claim 11,wherein said optical emitter emits light towards an eye of said viewer,whereas said optical detector detects property of said light reflectedfrom said eye including: dispersion of said reflected light, duration ofpulse of said reflected light, delay between pulses of said reflectedlight, and direction of said reflected light.
 15. The method accordingto claim 9, wherein said eye sensor is embedded in a see-throughsubstrate through which that said viewer is displayed said virtualscene.
 16. The method according to claim 15, wherein said see-throughsubstrate is in contact with cornea of an eye of said viewer.